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Abstract 


Exercise  Gamma  was  designed  to  be  a  complete  third-party  test  and  evaluation  of  the 
Collaborative  Capability  Definition,  Engineering  and  Management  (CapDEM)  approach.  The 
primary  goal  of  this  final  iteration  of  the  CapDEM  Evaluation  Strategy  was  to  test  and  evaluate 
the  CapDEM  approach  using  a  ‘real  problem’  based  on  a  departmentally-defined  scenario  and 
executed  by  a  team  composed  of  DND/CF  members.  The  intent  was  to  validate  the  necessary 
people,  process  and  materiel  to  address  a  ‘real  problem’  while  still  enabling  the  observation  and 
study  of  its  application  within  an  increasingly  operational-like  environment.  As  the  third  and 
final  exercise,  Exercise  Gamma  was  the  largest  and  most  ambitious  of  the  evaluation  efforts,  the 
result  of  gradual,  controlled  growth  from  one  exercise  to  another,  benefiting  from  the 
accumulated  experience  of  the  evaluation  team  along  with  its  validated  evaluation  strategy.  The 
exercise  shifted  away  from  incrementally  controlled  experimentation  towards  the  reality  of  actual 
departmental  clients  applying  Capability  Engineering  and  CapDEM’s  ability  to  meet  those  needs. 
Accordingly,  this  report  summarizes  the  results  of  Exercise  Gamma  which  was  undertaken  to 
evaluate  the  three  fundamental  axes  that  compose  the  Capability  Engineering  (CE)  construct 
(i.e.,  People,  Process  and  Materiel).  Specifically,  the  report  outlines  the  conduct  of  Exercise 
Gamma,  the  results  of  observations  and  focus  groups  that  were  conducted  throughout  the 
exercise,  and  provides  discussion  and  recommendations  to  consider  in  terms  of  the  potential 
institutionalization  of  the  CapDEM  approach  within  the  department. 


Resume 


L’exercice  Gamma  a  ete  concu  pour  etre  une  epreuve  et  une  evaluation  tout  a  fait  independante 
de  l’approche  axee  sur  la  definition,  l’ingenierie  et  la  gestion  collaboratives  des  capacites 
(DIGCap).  L’objectif  principal  de  ce  dernier  volet  de  la  Strategic  devaluation  de  l’approche 
DIGCap  consistait  a  mettre  a  l’essai  et  a  evaluer  cette  derniere  a  l’aide  d’un  «  probleme  reel » 
fonde  sur  un  scenario  defini  par  le  Ministere,  l’approche  etant  alors  mise  en  oeuvre  par  une  equipe 
composee  de  membres  du  MDN  et  des  FC.  L’ intention  etait  de  valider  les  personnes,  le 
processus  et  le  materiel  necessaires  pour  s’attaquer  a  un  «  probleme  reel »,  tout  en  permettant 
Tobservation  et  T  etude  de  l’application  du  processus  dans  un  contexte  a  caractere  de  plus  en  plus 
operationnel.  En  tant  que  le  troisieme  et  dernier  exercice,  1’exercice  Gamma  a  ete  le  plus  vaste  et 
le  plus  ambitieux  de  tous  les  efforts  devaluation;  il  resultait  de  la  croissance  graduelle  et 
controlee  s’etant  produite  d’un  exercice  a  l’autre  et  il  a  beneficie  de  T  experience  cumulative 
acquise  par  l’equipe  devaluation  et  de  sa  strategic  devaluation  validee.  L’Exercice  s’est  eloigne 
des  experiences  progressivement  controlees  pour  evoluer  vers  la  realite  de  veritables  clients 
ministeriels,  en  appliquant  l’ingenierie  des  capacites  et  l’outil  DIGCap  pour  repondre  aux  besoins 
de  ces  derniers.  Par  consequent,  le  present  rapport  resume  les  resultats  de  Texercice  Gamma,  qui 
a  ete  entrepris  pour  evaluer  les  trois  axes  fondamentaux  du  concept  structurel  de  l’ingenierie  des 
capacites  (IC)  :  les  personnes,  le  processus  et  le  materiel.  Le  rapport  decrit  T  execution  de 
Texercice  Gamma  et  les  resultats  des  observations  faites  pendant  T Exercice  et  des  interventions 
des  groupes  temoins;  il  offre  une  discussion  et  des  recommandations  a  etudier  relativement  a 
Tinstitutionnalisation  eventuelle  de  l’approche  DIGCap  au  sein  du  Ministere. 
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Executive  summary 


CapDEM  Exercise  Gamma:  Results  and  Discussion 

Wayne  Robbins;  Barbara  Waruszynski;  Claire  Lalancette;  Michel  Lizotte; 
Christophe  Necaille;  DRDC  Ottawa  TR  2011-044;  Defence  R&D  Canada  - 
Ottawa;  June  2011. 

Introduction 

Exercise  Gamma  was  intended  to  be  a  complete  third-party  test  and  evaluation  of  the 
Collaborative  Capability  Definition,  Engineering  and  Management  (CapDEM)  approach.  The 
primary  goal  of  this  final  iteration  of  the  CapDEM  Evaluation  Strategy  was  to  test  and  evaluate 
the  CapDEM  approach  as  applied  by  a  team  composed  of  DND/CF  members  on  their  own  ‘real 
problem’  based  on  a  departmentally-defined  scenario.  The  intent  was  to  validate  the  necessary 
people,  process  and  materiel  to  address  a  ‘real  problem’  while  still  enabling  the  observation  and 
study  of  its  application  within  an  increasingly  operational-like  environment.  Doing  so  involved 
capturing  the  experience  of  an  external  group  applying  the  process,  as  well  as  assessing  its 
readiness  and  those  issues  impacting  its  application. 

Exercise  Gamma  was  the  largest  and  most  ambitious  of  the  evaluation  efforts,  the  result  of 
gradual,  controlled  growth  from  one  exercise  to  another,  benefiting  from  the  accumulated 
experience  of  the  evaluation  team  along  with  its  validated  evaluation  strategy.  It  shifted  away 
from  incrementally  controlled  experimentation  towards  the  reality  of  actual  departmental  clients 
applying  Capability  Engineering  and  CapDEM’ s  ability  to  meet  those  needs. 

Results 

This  report  summarizes  the  results  of  Exercise  Gamma,  the  third  and  final  evaluation  exercise 
undertaken  to  evaluate  the  three  fundamental  axes  that  compose  the  Capability  Engineering  (CE) 
construct  (i.e.,  People,  Process  and  Materiel).  As  a  complete  third-party  test  and  evaluation,  it 
shifted  emphasis  from  an  ‘internal  team’  using  CapDEM  towards  the  reality  of  external  groups 
using  the  CapDEM  approach  to  address  their  own  problem  by  themselves. 

The  results  of  Exercise  Gamma  were  obtained  from  a  combination  of  various  focus  groups  as 
well  as  observations  of  the  Capability  Engineering  Team  (CET)  as  they  applied  the  Capability 
Engineering  Process  (CEP)  using  the  Collaborative  Engineering  Environment  (CEE).  In  terms  of 
each  axis,  a  synopsis  of  Exercise  Gamma’s  results  and  recommendations  is  as  follows: 

-  People  Axis 

Mandate  use  of  the  team  charter.  To  maintain  a  cohesive  and  functional  CET  with  an  effective 
understanding  and  fulfillment  of  its  roles  by  the  appropriate  individuals,  there  must  be  a  more 
defined  link  between  the  team  charter  and  the  day-to-day  governance  and  functioning  of  the  CET. 
There  needs  to  be  less  dependency  on  team  volition  to  use  the  charter  as  a  practical  and 
authoritative  guide,  and  not  just  use  it  as  an  ancillary  reference  for  the  alignment  of  roles  and 
individuals  to  specific  parts  of  the  process  (CEP)  as  well  as  the  technological  environment  (CEE). 
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Utilize  appropriately  designated  and  dedicated  personnel.  The  CET  must  be  composed  of 
suitably  skilled  and  interested  individuals  who  are  appropriately  available  and  in  sufficient 
number  according  to  the  requirements  of  the  CE  effort.  Participating  individuals  need  to  be 
dedicated  to  the  CET  and  not  divided  between  multiple  jobs  as  part  of  ensuring  consistent 
availability  and  fewer  conflicts  in  terms  of  time  and  accessibility.  Aptitude,  expertise,  availability 
and  appropriate  role  assignments  are  key  elements  of  a  successful  CET. 

Integrate  expertise  within  the  CET.  The  CET  needs  to  have  resident  expertise  not  only  on  the 
problem  space  being  addressed,  but  also  in  terms  of  the  CE  construct.  Knowledge  of  both  of  the 
CEP  and  the  CEE  would  help  members  see  the  ‘big  picture’  of  the  effort,  including  how  the  parts 
fit  together  and  how  they  aid  in  the  performance  of  various  tasks. 

Constructively  manage  CET  roles.  The  roles  on  the  CET  must  be  managed  in  a  concerted, 
clear  and  transparent  manner.  That  is,  the  CET  must  not  be  left  to  flounder  or  its  structure  be 
changed  in  an  ad  hoc  fashion.  To  ensure  cohesive  CET  behaviour,  it  is  important  not  to  create 
confusing  functional  overlap  (i.e.,  ensure  well-delineated  responsibilities)  or  distract  members 
through  interpersonal  issues  (e.g.,  ‘turf  war’)  that  can  have  adverse  effects  on  collaboration  and 
team  dynamics.  It  is  also  important  to  clarify  the  purpose  and  alignment  of  support  roles  as  part 
of  facilitating  good  working  relationships,  embedding  of  expertise  and  providing  appropriate 
inter-role  linkage  (including  the  knowledge  of  ‘who  to  go  to’).  The  selection  and  availability  of 
skilled  leadership  is  also  fundamental. 

Increase  and  clarify  alignment  between  CEE  and  CET.  Exercise  Gamma  illustrated  the  need 
for  a  more  effective  bridging  between  the  People  and  Materiel  axes.  An  improved  alignment 
between  the  CEE  and  CET  is  central  to  achieving  increased  self-sufficiency  and  higher 
productivity,  both  as  a  team  and  as  individual  members.  Specifically,  increased  self-confidence 
with  the  technology  would  enable  the  CET  to  effectively  and  assuredly  use  and  explore  novel 
application  of  the  CEE.  Consequently,  they  would  be  able  to  function  more  independently,  rather 
than  disrupt  their  workflow  by  requiring  continuous  interaction  with  outside  expertise.  Further, 
the  performance  of  individual  team  members  can  influence  interactions  amongst  the  rest  of  the 
CET,  thus  impacting  team  dynamics  and  consequently,  the  efficacy  of  their  teamwork  and 
collaboration. 

-  Process  Axis 

Formalize  identifiable  linkages  between  CEP  and  CBP.  Despite  their  existence,  the  linkages 
between  Capability -Based  Planning  (CBP)  and  the  CEP  were  not  easily  identified  by  the  CET 
members.  As  a  result,  the  CET  was  apprehensive  in  terms  of  how  to  proceed  in  a  coordinated 
manner  that  respected  both  processes.  Future  CEP  versions  must  formalize  such  linkages  and 
make  them  more  easily  identifiable;  indeed,  when  the  development  and  utilization  of  the  process 
are  under  the  purview  of  a  single  organization  (i.e.,  Chief  of  Force  Development  (CFD)),  such 
synchronization  will  become  more  straightforward. 

Clarify  CEP  lifecycle  and  progression.  Key  aspects  of  the  CEP,  such  as  its  iterative, 
incremental  and  multi-stage  design,  were  difficult  for  CET  to  understand;  specifically,  confusion 
between  the  aspects  and  how  they  related  to  each  other  was  encountered.  Clarity  of  the  CEP 
lifecycle,  including  the  relationship  and  progression  between  the  above  aspects  of  the  CEP  must 


IV 
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be  more  clearly  communicated.  In  the  broadest  sense,  the  CET  needs  a  clear  understanding  as 
how  to  execute,  manage  and  understand  stages  in  an  iterative  and  incremental  manner. 

Provide  alternative  CEP  specifications.  There  was  general  consensus  that  the  provision  of 
complementary  specifications  for  the  CEP,  specifically  the  use  of  deliverable-centric  and  task¬ 
centric  methods,  would  foster  greater  understanding  and  acceptance  of  the  process.  A  key 
challenge  of  such  a  dual-pronged  approach  will  be  to  maintain  coordination  and  consistency 
between  them. 

Ensure  workflow  independence.  Workflows  should  be  specified  suitably  orthogonal  to 
(i.e.,  independently  of)  the  specific  problem  and  technical  environment.  However,  they  must  be 
linked  in  an  illustrative  (normative)  manner  to  the  CEE  to  better  enable  CET  understanding. 

Clarify  CEP  and  CEE  linkage  relative  to  information  management  practices.  The  linkage 
and  mutual  influence  amongst  information  modelling,  tool  input  and  output,  process  input  and 
output  (i.e.,  deliverables)  and  their  relative  specification  needs  considerable  attention  and 
forethought.  Facilitating  productive  application  of  the  CEE  and  the  CEP  through  the  effective  use 
of  information  management  requires  the  use  of  well-founded  information  structuring  and 
management  principles.  Further,  such  principles  will  serve  as  the  basis  for  well-principled  and 
well-informed  exploration  of  said  structuring,  versus  potentially  negative  side  effects  resulting 
from  misinformed,  ad  hoc  changes. 

Investigate  deliverable  benchmarking  and  applicability  to  decision  making.  The  merit  of 
creating  benchmarks  for  deliverable  completion  and  how  to  provide  guidance  in  support  of  such 
benchmarking  remain  open  questions.  The  CET  was  able  to  illustrate  the  potential  of 
architectures  and  capability  engineering  within  the  force  development  process;  however,  they 
were  not  able  to  make  solid  conclusions  about  their  applicability  within  the  decision  process. 

Follow  the  process  and  accept  its  variability.  There  has  been  and  will  always  be  natural 
variance  in  the  conduct  of  the  CEP,  in  part  due  to  its  descriptive  rather  than  prescriptive 
specification.  While  the  amount  of  variance  in  Exercise  Gamma  was  regarded  as  atypical  based 
on  issues  of  process  maturity  and  practitioner  experience,  going  forward  it  will  be  important  to 
properly  control  process  evolution  during  its  application  (i.e.,  avoid  utilizing  multiple  versions  of 
the  process  within  a  single  instance).  It  will  also  be  important  to  limit  premature  workflow 
customization  as  part  of  avoiding  unnecessary  divergence  from  recommended  practices. 

-  Materiel  Axis 

Ensure  reliable  infrastructure  provisioning  and  management.  The  use  of  multiple  and 
independently  managed  networking  and  computing  infrastructures  was  an  unremitting  source  of 
user  difficulties  and  technical  challenges,  which  grew  relative  to  the  number  of  participants  and 
the  increased  breadth  of  organizations  represented.  Typical  difficulties  included  unplanned 
and/or  unannounced  network  policy  changes  that  adversely  affected  the  remote  use  of  tools 
outside  of  the  host  network.  The  consolidation  of  work  efforts  (i.e.,  use  of  the  CEE)  on  the  host 
network  would  eliminate  many  of  the  cross-network  issues.  While  external  access  remains 
desirable,  the  variety  and  configuration  of  access  points  needs  to  be  limited  and  well-controlled  as 
a  matter  of  pragmatism.  Any  such  changes  would  also  require  further  consideration  in  terms  of 
security  and  classification.  Indeed,  short  of  providing  cross-infrastructure  service  level 
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agreements,  the  unified  existence,  application,  management  and  support  of  the  CEE  on  a  single 
network  infrastructure  merits  thorough  consideration. 

Investigate  requirements  and  alternatives  for  secure  distributed  collaboration  using  non- 
homogenous  environments.  A  means  to  enable  and  support  different  internal  and  external 
configurations  of  the  classified  CEE  requires  further  study,  including  both  technical  and 
security/classification  issues.  While  the  value  of  such  functionality  is  likely  to  markedly  increase 
as  CEE  use  becomes  commonplace,  the  viability  and  provision  of  external  access  remains  a 
substantive  challenge,  both  technically  and  in  terms  of  regulatory  and  policy  issues. 

Increase  focus,  expertise  and  alignment  in  terms  of  information  management  and  associated 
technological  capability.  There  were  considerable  challenges  in  providing  a  clear,  flexible, 
extensible,  scalable,  comprehensive  and  understandable  means  to  use,  link  and  describe 
information  that  was  amenable  to  CET  usage,  CEP  application  and  CEE  representation. 
Specifically,  issues  of  conceptual  understanding  and  pragmatics  of  implementation  were 
problematic.  Underlying  the  situation  was  a  lack  of  suitable  CET  expertise,  Subject  Matter 
Expert  (SME)  availability  and  a  variety  of  technical  and  operational  issues.  Ultimately,  there  was 
(and  will  increasingly  be)  a  need  to  shift  towards  the  use  of  composable  (and  conceptually 
interoperable)  information  architectures. 

Facilitate  workflow  independence.  As  part  of  facilitating  learning  and  providing  a  baseline  for 
initial  use,  workflows  must  be  linked  in  an  illustrative  (normative)  manner  to  the  CEE;  however, 
such  linkages  must  be  done  so  as  to  avoid  unnecessarily  restrictive  limits  or  dependencies  on  the 
axes  as  they  mature  and  change  over  time.  That  is,  the  decoupling  of  logical  (process)  vs. 
physical  (technological)  workflows  creates  a  more  adaptable  way  to  repurpose  and  realign  the 
‘what’  from  the  ‘how’  in  a  more  granular  (and  therefore  flexible)  way.  Indeed,  the  move  towards 
composability  and  conceptual  interoperability  necessitates  that  workflows  be  specified  in  a 
suitably  orthogonal  manner  to  (i.e.,  independently  of)  a  specific  problem  and  technical 
environment.  Facilitating  workflow  independence  from  a  CEE  perspective  is  also  a  necessary 
complement  to  realize  process-centric  workflow  independence. 

Increase  support  for  interfaces  to  external  processes.  As  part  of  achieving  clearer  linkage 
between  the  CEP  to  external  processes,  there  is  a  need  to  investigate  the  technological 
requirements  and  implications  of  linking  CEE  systems  to  those  used  by  external  processes.  That 
is,  this  issue  speaks  to  the  need  for  technological  support  along  the  CEP  interface  to  other 
departmental  processes.  In  working  towards  this  capability,  clearer  identification  of  relevant 
organizations,  processes  and  systems  will  need  to  be  provided,  along  with  the  consideration  of 
issues  such  as  representation,  compatibility,  security  and  access  (e.g.,  permissions). 

Clarify  CEE  support  for  information  management  practices.  Appropriate  structuring  and  use 
of  information  management  principles  in  terms  of  the  CEE  need  to  be  applied  at  the  interface 
between  axes  as  part  of  providing  a  composable  interface.  Considerable  forethought  will  be 
required  to  facilitate  productive  process  application  given  the  confluence  of  information 
modelling,  tool  input  and  output,  process  input  and  output  and  their  use  relative  to  each  other. 
Indeed,  to  avoid  only  superficial  interaction  between  information  management  practices  and  the 
technological  environment,  well-founded  information  structuring  and  management  principles 
need  to  be  applied  and  serve  as  the  basis  for  well-principled  application  of  the  CEE. 


VI 
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Increase  and  clarify  alignment  between  CEE  and  other  axes.  Ensuring  appropriate 
application  of  the  CEE  by  the  CET  is  a  key  element  of  their  ability  to  function  independently  and 
with  fewer  workflow  disruptions.  Consequently,  ensuring  appropriate  technical  propensity  along 
with  a  balance  of  technology  cohesion  and  delineation  would  be  part  of  achieving  clear  alignment 
to,  and  effective  usage  of,  the  CEE.  Such  clarity  would  reduce  the  potential  of  inappropriate 
technology  application,  and  help  focus  the  CET  relative  to  useful  and  forward-looking 
technologies.  In  as  much  as  the  above  issues  can  impact  the  performance  of  individual  members, 
it  can  transitively  impact  team  dynamics,  and  therefore  the  efficacy  of  their  teamwork. 
Consequently,  the  broader  engineering  effort  could  then  be  significantly  affected,  either  positively 
or  negatively. 

Integrate  CEE  expertise  within  the  CET.  The  availability  of  CEE  expertise  within  the  CET  is 
useful  both  as  part  of  addressing  tool  application  and  information  management  issues  but  also  to 
facilitate  awareness  of  potential  application,  advocacy  of  suitable  usage  and/or  which  pitfalls  to 
avoid  (i.e.,  to  provide  mentorship  and  coaching).  Complementary  resident  expertise  in  terms  of 
the  CE  construct  would  aid  in  creating  a  holistic  understanding  and  reduce  the  potential  for 
technological  silos  (for  example,  knowledge  of  particular  software  but  a  lack  of  awareness  in 
terms  of  its  implications  within  the  broader  technological  environment  and  the  engineering  effort 
itself). 

Promote  increased  understanding  and  usability.  The  themes  of  understanding  and  usability 
underlined  many  of  the  issues  that  affected  CEE  use.  In  particular,  how  particular  tools  should  be 
used  both  individually  and  in  conjunction  with  each  other  had  significant  impact  on  the  CET 
members’  work  efforts.  By  addressing  the  issues  of  expertise  integration  and  improved 
clarity/alignment,  individual  members  can  be  more  focused,  which  will  also  facilitate  the 
provision  of  training,  mentoring  and  coaching,  along  with  a  stronger  ability  to  target  problematic 
areas  in  terms  of  technical  support  and  CEE  evolution. 

Expand  technology  and  associated  capability  base.  There  is  an  ongoing  need  to  explore 
alternative  and  developing  technologies  as  part  of  providing  an  innovative  and  creative 
collaborative  engineering  environment.  Such  a  ‘technology  watch’  will  prove  essential  in 
addressing  the  advancing  technological  landscape  in  conjunction  with  the  changing  breadth  and 
depth  of  tool  functionality,  the  evolving  needs  of  the  department  and  a  growing,  more  capable 
user  base. 

Significance 

As  the  final  exercise,  Exercise  Gamma  was  the  culmination  of  gradual,  controlled  growth  from 
one  exercise  to  another,  facilitating  increased  credibility  in  the  Evaluation  Team  and  its  strategy 
along  with  the  ability  to  more  definitively  examine  the  issue  of  scalability  in  the  application  of  the 
CapDEM  axes.  The  exercise  enabled  the  collection  of  end  user  (operator)  feedback  so  as  to 
further  refine  each  axis  through  application  of  the  Capability  Engineering  construct,  allowing 
continuous  improvement  of  the  construct  during  its  development.  Future  application  of  the 
CapDEM  approach  outside  of  the  Technology  Demonstration  Programme  (TDP)  will  be  well 
served  by  the  veracity  of  Exercise  Gamma  as  well  as  the  comparison  across  the  whole  series  of 
evaluation  exercises. 
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Future  plans 


The  strategic  intent  of  Exercise  Gamma  was  to  understand  and  assess  the  issues  and  implications 
of  utilizing  Capability  Engineering  in  an  external  context,  specifically  by  identifying  those 
outstanding  issues  that  could  impact  its  way  forward.  This  final  exercise  was  intended  to  serve  as 
the  basis  for  further  evolution  and  application  of  the  CapDEM  approach  as  it  becomes 
transitioned  to  the  larger  department  through  the  force  development  community.  Consequently, 
discussions  and  recommendations  for  the  way  ahead  are  put  forward  in  this  light. 

Given  that  the  application  of  systems  engineering  at  a  capability  level  is  new  to  the  department, 
further  evolution  and  integration  of  the  approach  into  departmental  practices  will  be  well-served 
by  additional  and  incremental  application  and  experimentation  in  a  variety  of  circumstances. 
Notably,  to  effectively  enable  institutionalization,  the  creation  of  specially  trained  ‘CE  officers’  is 
proposed  to  facilitate  the  practice  of  Capability  Engineering  through  the  availability  of 
knowledgeable  and  experienced  practitioners.  Furthermore,  continuous  improvement  of  the  CE 
approach  is  necessary  as  CE  evolves  into  its  niche  within  the  force  development  community. 
Moreover,  encompassing  all  of  these  aspects  is  the  challenge  of  institutional  resistance  to  change, 
combined  with  the  difficulty  of  obtaining  knowledgeable  personnel  that  can  be  fully  dedicated  to 
the  effort  at  hand. 

The  level  of  detail  and  volume  of  analytical  products  required  to  satisfy  capability-level  decisions 
will  not  completely  be  answered  until  a  use  case  has  its  output  transitioned  into  implementation  in 
the  capability  production  domain.  However,  Exercise  Gamma’s  analytical  products  were  well 
received  by  the  exercise  participants  as  well  as  its  sponsors  which  represented  key  organizations 
related  to  the  eventual  institutionalization  of  the  CapDEM  approach.  Exercise  Gamma  can 
therefore  be  regarded  as  a  positive  step  in  bridging  the  transition  of  Capability  Engineering  from 
its  research  and  development  roots  to  exploitation  by  the  force  development  community. 
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Introduction 

L’exercice  Gamma  a  ete  coniju  pour  etre  une  epreuve  et  une  evaluation  tout  a  fait  independantes 
de  l’approche  axee  sur  la  definition,  l’ingenierie  et  la  gestion  collaboratives  des  capacites 
(DIGCap).  L’objectif  principal  de  ce  dernier  volet  de  la  Strategic  devaluation  de  l’approche 
DIGCap  consistait  a  mettre  a  l’essai  et  a  evaluer  cette  derniere  a  l’aide  d’un  «  probleme  reel » 
fonde  sur  un  scenario  defini  par  le  Ministere,  l’approche  etant  alors  mise  en  oeuvre  par  une  equipe 
composee  de  membres  du  MDN  et  des  FC.  L’ intention  etait  de  valider  les  personnes,  le 
processus  et  le  materiel  necessaires  pour  s’attaquer  a  un  «  probleme  reel  »,  tout  en  permettant 
l’observation  et  l’etude  de  l’application  du  processus  dans  un  contexte  a  caractere  de  plus  en  plus 
operationnel.  Pour  ce  faire,  il  a  fallu  recueillir  les  experiences  d’un  groupe  de  l’exterieur 
appliquant  le  processus  et  evaluer  son  etat  de  preparation  et  les  questions  influant  sur  son 
application. 

L’exercice  Gamma  a  ete  le  plus  vaste  et  le  plus  ambitieux  de  tous  les  efforts  devaluation;  il 
resultait  de  la  croissance  graduelle  et  controlee  s’etant  produite  d’un  exercice  a  l’autre  et  il  a 
beneficie  de  l’experience  cumulative  acquise  par  l’equipe  devaluation  et  de  sa  strategic 
devaluation  validee.  L’Exercice  s’est  eloigne  des  experiences  progressivement  controlees  pour 
evoluer  vers  la  realite  de  veritables  clients  ministeriels,  en  appliquant  l’ingenierie  des  capacites  et 
l’outil  DIGCap  pour  repondre  aux  besoins  de  ces  derniers. 

Resultats 

Le  present  rapport  resume  les  resultats  de  l’exercice  Gamma,  troisieme  exercice  entrepris  pour 
evaluer  les  trois  axes  fondamentaux  du  concept  structurel  de  l’ingenierie  des  capacites  (IC)  :  les 
personnes,  le  processus  et  le  materiel.  En  tant  qu’epreuve  et  evaluation  tout  a  fait  independantes, 
l’Exercice  a  moins  mis  1’ accent  sur  une  «  equipe  interne  »  utilisant  l’approche  DIGCap  et  il  a 
plutot  favorise  la  realite  des  groupes  de  l’exterieur  recourant  a  l’approche  pour  regler  eux-memes 
leurs  propres  problemes. 

Pour  obtenir  les  resultats  de  l’exercice  Gamma,  nous  avons  eu  recours  a  des  groupes  temoins  et  a 
des  observations  faites  par  l’Equipe  d’ingenierie  des  capacites  (EIC)  tandis  qu’elle  appliquait  le 
processus  d’ingenierie  des  capacites  (PIC)  a  l’aide  de  l’environnement  d’ingenierie  des  capacites 
(EnvIC).  Voici  un  resume  des  resultats  de  l’Exercice  et  des  recommandations  formulees  a  l’issue 
de  ce  dernier,  relativement  a  chacun  des  axes  : 
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-  L’axe  des  personnes 


Utilisation  obligatoire  de  la  charte  d’equipe.  Afin  de  conserver  a  l’EIC  sa  cohesion  et  sa 
fonctionnalite  et  d’amener  les  personnes  concernees  a  comprendre  efficacement  ses  roles  et  a  bien 
les  remplir,  il  doit  y  avoir  un  lien  plus  defini  entre  la  charte  d’equipe,  d’une  part,  et,  d’autre  part, 
la  regie  et  le  fonctionnement  quotidiens  de  l’EIC.  L’equipe  doit  etre  plus  encline  a  utiliser  la 
charte  a  titre  de  guide  pratique  faisant  autorite,  plutot  que  comme  un  ouvrage  de  reference 
auxiliaire,  pour  ce  qui  est  d’ aligner  les  roles  et  les  personnes  sur  des  elements  particuliers  du 
processus  (PIC)  et  sur  l’environnement  technologique  (EnvIC). 

Employer  un  personnel  devoue  et  bien  choisi.  L’EIC  doit  se  composer  de  personnes 
interessees  par  le  projet  et  possedant  les  competences  voulues;  elles  doivent  etre  suffisamment 
disponibles  et  nombreuses,  tout  dependant  des  exigences  de  1’ effort  a  deployer  en  matiere  d’IC. 
Les  participants  doivent  se  consacrer  entierement  a  l’EIC,  au  lieu  de  se  diviser  entre  de  multiples 
taches,  de  maniere  a  garantir  une  disponibilite  constante  et  a  reduire  le  nombre  de  conflits 
d’horaire  et  d’ accessibility  Les  aptitudes,  les  competences,  la  disponibilite  et  l’attribution 
judicieuse  des  roles,  voila  autant  d’elements  cles  dont  depend  le  succes  de  l’EIC. 

Integrer  les  competences  dans  l’EIC.  L’EIC  a  besoin  de  posseder  en  son  sein  des 
connaissances  non  seulement  sur  l’espace  faisant  probleme  mais  aussi  sur  le  concept  structurel 
d’IC.  La  connaissance  du  PIC  et  de  1’EnvIC  aiderait  les  membres  a  voir  l’ensemble  de  la 
situation,  y  compris  comment  les  elements  s’agencent  entre  eux,  et  elle  favoriserait  1’ execution 
des  diverses  taches. 

Gerer  d’une  faqon  constructive  les  roles  de  l’EIC.  Les  roles  de  l’EIC  doivent  etre  geres  d’une 
faqon  concertee,  claire  et  transparente.  En  d’autres  mots,  il  ne  faut  pas  laisser  l’EIC  errer  au 
hasard  ou  modifier  sa  structure  de  faqon  aleatoire.  Afin  d’assurer  la  cohesion  de  l’EIC  dans  son 
comportement,  il  importe  de  n’engendrer  aucun  recoupement  fonctionnel  pretant  a  confusion 
(veiller  a  ce  que  les  responsabilites  soient  bien  definies)  et  de  veiller  a  ce  que  les  membres  ne 
soient  pas  distraits  par  des  conflits  interpersonnels  risquant  d’ avoir  des  effets  nuisibles  sur  la 
collaboration  et  la  dynamique  collective.  Il  importe  aussi  de  clarifier  le  but  et  l’alignement  des 
roles  de  soutien  afin  de  faciliter  les  bonnes  relations  de  travail,  l’integration  des  competences  et 
l’etablissement  de  liens  utiles  entre  les  roles  (notamment,  connaitre  les  relations  hierarchiques). 
En  outre,  il  est  fondamental  de  choisir  des  chefs  competents  et  accessibles. 

Accroitre  et  clarifier  l’alignement  entre  1’EnvIC  et  l’EIC.  L’exercice  Gamma  a  montre  la 
necessite  d’etablir  un  lien  plus  efficace  entre  les  axes  «  Personnes  »  et  «  Materiel  ».  Un  meilleur 
alignement  entre  1’EnvIC  et  l’EIC  est  essentiel  afin  de  favoriser  un  accroissement  de  l’autonomie 
et  de  la  productivite  de  l’equipe  et  de  ses  membres.  Plus  precisement,  si  l’EIC  se  sentait  plus  sure 
d’elle-meme  sur  le  plan  technologique,  elle  pourrait  mieux  utiliser  et  explorer  des  applications 
novatrices  de  1’EnvIC.  Par  consequent,  les  membres  pourraient  fonctionner  avec  plus 
d’independance,  au  lieu  de  bouleverser  leur  cadence  de  travail  en  devant  recourir  constamment  a 
des  experts  de  l’exterieur.  En  outre,  le  rendement  de  membres  individuels  de  l’equipe  peut 
influer  sur  les  rapports  entre  les  autres,  ce  qui  a  des  consequences  pour  la  dynamique  collective 
et,  partant,  pour  l’efficacite  du  travail  d’equipe  et  de  la  collaboration. 
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-  L’AXE  DU  PROCESSUS 


Regulariser  des  liens  identifiables  entre  la  PFC  et  le  PIC.  Malgre  leur  existence,  les  liens 
entre  la  planification  fondee  sur  les  capacites  (PFC)  et  le  PIC  n’ont  pas  ete  reperes  facilement  par 
les  membres  de  l’EIC.  Par  consequent,  celle-ci  ne  savait  pas  trop  comment  coordonner  les  choses 
d’une  fatjon  qui  respecterait  les  deux  processus.  A  l’avenir,  les  versions  du  PIC  devront 
regulariser  ces  liens  et  les  rendre  plus  faciles  a  reperer;  en  fait,  quand  la  mise  au  point  et 
l’utilisation  d’un  tel  processus  releveront  d’une  seule  organisation  [p.  ex.,  le  Chef  - 
Developpement  des  Forces  (CDF)],  pareille  synchronisation  deviendra  plus  directe. 

Clarifier  le  cycle  de  vie  et  la  progression  du  PIC.  L’EIC  a  eu  du  mal  a  comprendre  certains 
aspects  du  PIC,  tels  que  sa  conception  iterative,  progressive  et  a  stades  multiples;  plus 
precisement,  elle  a  eu  de  la  difficulte  a  faire  la  distinction  entre  les  aspects  et  la  fa5on  dont  ils  se 
rapportent  l’un  a  1’ autre.  II  importe  de  mieux  expliquer  le  cycle  de  vie  du  PIC,  y  compris  la 
relation  entre  les  aspects  susmentionnes  et  la  progression  de  l’un  a  l’autre.  Au  sens  le  plus  large, 
l’EIC  doit  comprendre  clairement  comment  executer,  gerer  et  franchir  les  stades  d’une  faijon 
iterative  et  progressive. 

Fournir  des  specifications  de  rechange  pour  le  PIC.  Tous  s’entendaient  pour  dire  que  des 
specifications  complementaires  pour  le  PIC,  en  particulier  l’utilisation  de  methodes  axees  sur  les 
resultats  attendus  et  sur  les  taches,  accroitrait  la  comprehension  et  1’ acceptation  du  processus.  Le 
defi  principal  d’une  telle  demarche  a  deux  axes  consistera  a  conserver  la  coordination  et 
l’uniformite  entre  les  deux. 

Assurer  l’independance  des  flux  de  travail.  II  faut  definir  les  flux  de  travail  d’une  faqon  qui  se 
rapporte  directement  au  probleme  particulier  et  a  l’environnement  technique.  Cependant,  ils 
doivent  etre  lies  d’une  faijon  indicative  (normative)  a  1’EnvIC  afin  de  favoriser  davantage  la 
comprehension  des  choses  par  l’EIC. 

Clarifier  les  liens  entre  le  PIC  et  1’EnvIC  par  rapport  aux  pratiques  de  gestion.  Les  rapports 
et  l’influence  mutuelle  observes  entre  les  modeles  d’information,  les  outils  fournis  et  produits,  les 
intrants  et  les  extrants  des  processus  (ex.  :  resultats  escomptes)  et  leurs  specifications  relatives 
necessitent  une  attention  considerable  et  une  reflexion  approfondie.  Afin  de  faciliter  1’ application 
productive  de  1’EnvIC  et  du  PIC  par  le  biais  de  l’utilisation  efficace  de  la  gestion  de 
1’ information,  il  faut  employer  des  principes  solides  de  gestion  et  de  structuration  de 
1’ information.  En  outre,  ces  principes  serviront  de  base  a  l’exploration  reflechie  et  judicieuse  des 
structures  en  question,  ce  qui  evitera  les  effets  secondaires  eventuels  que  produiraient  des 
changements  ponctuels  axes  sur  des  renseignements  errones. 

Mener  des  recherches  sur  les  criteres  d ’evaluation  des  resultats  escomptes  et  sur  leur 

applicabilite  au  processus  decisionnel.  Le  merite  qu’il  y  a  a  creer  des  points  de  reference  quant 
a  l’obtention  des  resultats  escomptes  et  la  question  de  savoir  comment  fournir  des  parametres 
guides  pour  appuyer  1’ adoption  de  ces  points  de  reference  demeurent  des  themes  que  l’on 
continue  de  debattre.  L’EIC  a  pu  faire  voir  le  potentiel  des  architectures  et  de  l’ingenierie  des 
capacites  au  sein  du  processus  de  developpement  des  forces,  mais  elle  n’a  pas  reussi  a  formuler 
des  conclusions  solides  sur  leur  applicabilite  dans  le  processus  decisionnel. 
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Suivre  le  processus  et  accepter  sa  variability.  L’ application  du  PIC  a  toujours  comporte  et 
comportera  toujours  une  variance  naturelle,  notamment  a  cause  de  la  specification  descriptive 
plutot  que  normative  du  processus.  L’ampleur  de  la  variance  dans  l’exercice  Gamma  a  ete 
consideree  comme  etant  atypique,  compte  tenu  de  la  maturite  du  processus  et  de  1’ experience  des 
praticiens,  mais  a  l’avenir,  il  importera  de  controler  convenablement  1’ evolution  du  processus 
pendant  son  application  (ex.  :  eviter  d’employer  de  multiples  versions  du  processus  dans  un  seul 
et  meme  cas).  II  importera  aussi  de  limiter  1’ adaptation  prematuree  des  flux  de  travail  pour  eviter 
ainsi  la  divergence  inutile  par  rapport  aux  pratiques  recommandees. 

-  L’axe  du  materiel 

Assurer  la  gestion  et  la  prestation  de  services  d ’infrastructure  fiables.  L’utilisation 
d’infrastructures  de  reseautage  et  de  calcul  multiples  et  gerees  independamment  a  ete  une  source 
constante  de  difficultes  et  de  defis  techniques  pour  les  utilisateurs,  difficultes  et  defis  qui 
augmentaient  en  fonction  du  nombre  de  participants  et  de  la  taille  des  organisations  representees. 
Parmi  les  difficultes  typiques,  il  y  avait  la  suivante  :  des  changements  non  planifies  et/ou  non 
annonces  dans  les  politiques  des  reseaux,  changements  qui  nuisaient  a  P  utilisation  des  outils  en 
dehors  du  reseau  hote.  Le  regroupement  des  efforts  (c’est-a-dire  le  recours  a  1’EnvIC)  dans  le 
reseau  hote  eliminerait  bon  nombre  des  problemes  dus  a  la  structure  interreseaux.  L’acces  depuis 
l’exterieur  demeure  souhaitable,  mais  il  faut  limiter  et  controler  judicieusement  la  variete  et  la 
configuration  des  points  d’acces,  pour  des  raisons  pragmatiques.  Afin  d’operer  n’importe  quel 
changement  du  genre,  il  faudrait  aussi  prendre  en  consideration  la  securite  et  la  classification.  En 
effet,  a  moins  d’etablir  des  accords  sur  les  niveaux  de  service  applicables  entre  les  diverses 
infrastructures,  l’existence,  1’ application,  la  gestion  et  le  soutien  de  1’EnvIC  au  sein  d’une  seule 
infrastructure  de  reseau  meritent  un  examen  approfondi. 

Mener  des  recherches  sur  les  besoins  dans  le  domaine  de  la  collaboration  decentralisee  et 
securisee  avec  des  environnements  non  homogenes  et  sur  les  solutions  de  rechange  a  cet 
egard.  Il  faut  etudier  davantage  un  moyen  de  dynamiser  et  de  soutenir  des  configurations 
internes  et  externes  differentes  de  1’EnvIC  classifie,  y  compris  les  questions  techniques  ou 
relatives  a  la  securite/classification.  Certes,  la  valeur  d’une  telle  fonctionnalite  augmentera  sans 
doute  a  mesure  que  l’emploi  de  1’EnvIC  deviendra  plus  courant,  mais  la  viabilite  et  l’offre  d’un 
acces  de  l’exterieur  continuent  de  representer  un  defi  de  taille  sur  le  plan  technique  et  du  point  de 
vue  de  la  reglementation  et  des  politiques. 

Accroitre  le  ciblage,  l’expertise  et  l’alignement  de  la  gestion  de  l’information  et  des 
capacites  technologiques  connexes.  Fournir  un  moyen  clair,  souple,  extensible,  evolutif, 
complet  et  comprehensible  pour  utiliser,  integrer  et  decrire  l’information  se  pretant  a  l’utilisation 
par  l’EIC,  a  l’application  du  PIC  et  a  la  representation  de  1’EnvIC  a  comporte  des  defis  de  taille. 
Plus  precisement,  la  comprehension  des  concepts  et  les  aspects  pragmatiques  de  la  mise  en  oeuvre 
ont  fait  probleme.  Il  y  avait  aussi  divers  elements  sous-jacents  :  un  manque  d’ expertise  suffisante 
dans  l’EIC,  la  disponibilite  d’experts  des  questions  abordees  et  toute  une  gamme  de  problemes 
techniques  et  operationnels.  En  fin  de  compte,  on  avait  besoin  (et  ce  besoin  ira  en  augmentant)  de 
passer  a  P utilisation  d’ architectures  de  l’information  composables  (et  interoperables  sur  le  plan 
conceptuel). 

Faciliter  l’independance  des  flux  de  travail.  Afin  de  faciliter  1’ apprentissage  et  d’etablir  un 
point  de  reference  pour  P  utilisation  initiale  des  flux  de  travail,  il  faut  creer  des  liens  indicatifs 
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(normatifs)  entre  eux  et  1’EnvIC;  cependant,  il  faut  le  faire  d’une  maniere  qui  evitera  les  limites 
inutilement  contraignantes  ou  les  dependances  par  rapport  aux  axes  a  rnesure  qu’ils  prendront  de 
la  maturite  et  qu’ils  evolueront  avec  le  temps.  En  d’autres  mots,  le  decouplage  des  flux  logiques 
(processus)  et  physiques  (technologiques)  favorise  avec  plus  d’adaptabilite  l’isolement  du 
«  quoi  »  par  rapport  au  «  comment  »  et  permet  de  lui  donner  un  nouveau  but  au  sein  de  petits 
groupes  (et,  partant,  accroitre  la  souplesse  du  processus).  En  effet,  pour  passer  a  une  composition 
interessante  et  a  l’interoperabilite  conceptuelle,  il  faut  que  les  flux  de  travail  soient  definis 
directement  et  independamment  par  rapport  a  un  probleme  particulier  et  a  un  environnement 
technique.  Il  faut  egalement  faciliter  l’independance  des  flux  de  travail  par  rapport  a  1’EnvIC, 
afin  de  faire  complement  a  l’independance  des  flux  de  travail  axes  sur  les  processus. 

Soutenir  davantage  les  interfaces  avec  les  processus  exterieurs.  Afin  de  clarifier  davantage 
les  liens  entre  le  PIC  et  les  processus  exterieurs,  il  importe  d’ analyser  les  exigences 
technologiques  et  les  consequences  de  l’etablissement  de  liens  entre  les  systemes  de  1’EnvIC  et 
ceux  employes  par  les  processus  exterieurs.  Autrement  dit,  nous  parlons  ici  de  la  necessite  d’un 
soutien  technologique  au  profit  de  l’interface  entre  le  PIC  et  d’autres  processus  ministeriels.  En 
travaillant  vers  la  creation  de  cette  capacite,  on  devra  designer  plus  clairement  les  organisations, 
les  processus  et  les  systemes  concernes,  tout  en  prenant  en  consideration  les  questions  de 
representation,  de  compatibility  de  securite  et  d’acces  (ex.  :  permissions). 

Clarifier  le  soutien  accorde  par  1’EnvIC  aux  pratiques  de  gestion  de  l’information.  Il  faut 
structurer  et  utiliser  convenablement,  a  l’interface  entre  les  axes,  les  principes  de  gestion  de 
l’information  relativement  a  1’EnvIC  si  l’on  veut  fournir  une  interface  composable.  Beaucoup  de 
prevoyance  s’imposera  pour  faciliter  l’application  productive  des  processus,  etant  donne  la 
confluence  des  modeles  d’ information,  des  outils  et  des  processus  utilises  et  produits  et  de  leur 
emploi  les  uns  par  rapport  aux  autres.  En  effet,  afin  d’eviter  une  interaction  superficielle  entre  les 
pratiques  de  gestion  de  l’information  et  1’ environnement  technologique,  nous  devrons  appliquer 
de  solides  principes  de  structuration  et  de  gestion  de  l’information  et  nous  en  servir  comme  d’un 
fondement  pour  1’ application  judicieuse  de  1’EnvIC. 

Renforcer  et  clarifier  l’alignement  entre  1’EnvIC  et  les  autres  axes.  Il  est  essentiel  de  garantir 
l’application  appropriee  de  1’EnvIC  par  les  membres  de  l’EIC,  pour  que  ceux-ci  puissent 
fonctionner  independamment  et  pour  reduire  le  nombre  de  perturbations  dans  les  flux  de  travail. 
Par  consequent,  afin  de  realiser  un  alignement  clair  sur  1’EnvIC  et  l’utilisation  efficace  de  ce 
dernier,  il  importerait  de  garantir  la  propension  technique  des  participants  ainsi  qu’un  equilibre 
entre  la  cohesion  et  le  groupement  des  technologies.  Pareille  clarte  reduirait  l’eventualite  d’une 
application  impropre  de  la  technologie  et  aiderait  a  orienter  l’EIC  vers  les  technologies  utiles  et 
tournees  vers  l’avenir.  Dans  la  mesure  ou  les  questions  susmentionnees  peuvent  influer  sur  le 
rendement  des  divers  membres  de  l’Equipe,  elles  risquent  aussi  d’influer  transitivement  sur  la 
dynamique  de  cette  derniere  et,  partant,  sur  la  qualite  de  son  travail  collectif.  L’effort 
d’ingenierie  global  pourrait  done  etre  touche  lui  aussi,  soit  positivement,  soit  negativement. 

Integrer  l’expertise  sur  1’EnvIC  dans  l’EIC.  La  presence  d’une  expertise  sur  1’EnvIC  dans 
l’EIC  est  utile  pour  s’attaquer  aux  questions  relatives  a  l’application  des  outils  et  a  la  gestion  de 
l’information  et  aussi  pour  faciliter  la  prise  de  conscience  d’ applications  possibles,  la  mise  en 
exergue  d’utilisations  appropriees  et/ou  la  mise  en  garde  contre  les  ecueils  a  eviter  (en  d’autres 
mots,  pour  fournir  un  mentorat  et  un  encadrement).  Une  expertise  complementaire  presente  sur 
place  relativement  au  concept  structurel  de  l’IC  aiderait  a  creer  une  comprehension  holistique  et  a 
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reduire  1’ apparition  eventuelle  d’un  cloisonnement  technologique  (par  exemple,  la  connaissance 
d’un  logiciel  donne,  mais  l’ignorance  de  ses  implications  dans  l’environnement  technologique 
global  et  au  sein  de  1’ effort  d’ingenierie  meme). 

Promouvoir  la  comprehension  et  l’utilite.  Les  themes  de  la  comprehension  et  de  l’utilite  ont 
fait  partie  integrante  de  bon  nombre  de  questions  relatives  a  1’ application  de  1’EnvIC.  En 
particulier,  la  faqon  dont  des  outils  donnes  devraient  etre  employes  individuellement  et  les  uns 
avec  les  autres  a  eu  un  effet  considerable  sur  les  efforts  des  membres  de  l’EIC.  En  se  penchant 
sur  1’ integration  de  1’ expertise  et  sur  1’ amelioration  de  la  clarte  et  de  l’alignement,  on  favorisera 
une  meilleure  concentration  des  efforts  des  membres  de  l’equipe,  ce  qui  facilitera  aussi  la 
formation,  le  mentorat  et  l’encadrement  ainsi  qu’une  capacite  accrue  de  s’attaquer  aux  secteurs 
problematiques,  quant  au  soutien  technique  et  a  1’evolution  de  1’EnvIC. 

Elargir  la  base  technologique  et  la  gamme  des  capacites  connexes.  II  faut  constamment 
explorer  les  technologies  de  rechange  et  celles  qui  sont  en  devenir  afin  de  susciter  un  contexte  de 
l’ingenierie  novateur  et  creatif  axe  sur  la  collaboration.  Une  telle  «  veille  technologique  » 
s’averera  essentielle  face  a  1’evolution  du  paysage  technologique  et  de  la  fonctionnalite  des  outils, 
toujours  plus  large  et  profonde,  aux  besoins  en  devenir  du  Ministere  et  a  1’  accroissement  du 
bassin  d'utilisateurs  et  de  leurs  capacites. 

Importance 

Comme  l’exercice  Gamma  correspondait  au  dernier  stade  du  projet,  il  a  ete  le  point  culminant 
d’une  croissance  graduelle  et  controlee  d’un  stade  a  l’autre,  ce  qui  a  accru  la  credibilite  des 
membres  de  l’equipe  devaluation  et  de  sa  strategie  ainsi  que  sa  capacite  d’examiner  d’une  faqon 
plus  definitive  la  question  de  l’extensibilite  dans  1’ application  des  axes  DIGCap.  L’Exercice  a 
permis  de  recueillir  les  opinions  des  utilisateurs  (operateurs)  finaux  et  de  parfaire  ainsi  chacun  des 
axes  grace  a  l’application  du  concept  structural  de  l’ingenierie  des  capacites  (IC),  ce  qui  a 
favorise  1’ amelioration  continue  du  concept  pendant  sa  mise  au  point.  A  l’avenir,  1’ application  de 
la  demarche  DIGCap,  en  dehors  du  Programme  de  demonstration  de  technologies  (PDT), 
profitera  de  la  veracite  de  l’exercice  Gamma  et  de  la  comparaison  faite  a  tous  les  stades  de  la 
serie  des  exercices  devaluation. 

Perspectives 

Le  but  strategique  de  l’exercice  Gamma  etait  de  comprendre  et  d’evaluer  les  aspects  et  les 
consequences  de  l’utilisation  de  l’ingenierie  des  capacites  dans  un  contexte  exterieur,  plus 
precisement  en  cernant  les  problemes  non  resolus  qui  risquaient  d’influer  sur  sa  progression.  Cet 
exercice  final  visait  a  servir  de  fondement  pour  1’evolution  et  l’application  futures  de  la  demarche 
DIGCap,  alors  qu’elle  fera  une  transition  a  P ensemble  du  Ministere,  par  1’ intermediate  des 
milieux  s’occupant  du  developpement  des  Forces.  Par  consequent,  les  discussions  et  les 
recommandations  relatives  a  l’avenir  se  situent  dans  ce  contexte. 

Comme  l’application  de  la  systemique  au  niveau  des  capacites  est  nouvelle  dans  le  Ministere,  on 
aura  avantage  a  faire  d’ autres  applications  et  experiences  graduelles  dans  des  circonstances 
diverses  pour  faire  evoluer  la  demarche  et  l’integrer  dans  les  pratiques  du  Ministere.  En 
particulier,  afin  de  favoriser  efficacement  l’institutionnalisation  de  la  demarche,  nous 
recommandons  de  creer  des  postes  d’agents  de  l’IC  et  de  leur  fournir  une  formation  speciale; 
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ainsi,  grace  a  leurs  connaissances  et  a  leur  experience,  ils  faciliteront  l’application  de  l’ingenierie 
des  capacites.  En  outre,  il  faudra  continuer  d’ameliorer  la  demarche  axee  sur  l’IC,  a  mesure  que 
celle-ci  evoluera  dans  son  creneau,  dans  les  milieux  du  developpement  des  Forces.  Par  ailleurs, 
chapeautant  tous  ces  aspects,  il  y  a  le  defi  de  la  resistance  des  institutions  au  changement 
conjuguee  a  la  difficulty  d’obtenir  un  personnel  competent  pouvant  se  consacrer  a  fond  au  travail 
a  accomplir. 

On  ne  saura  pas  au  juste  quels  produits  analytiques  seront  necessaries  pour  la  prise  des  decisions 
relatives  aux  capacites  tant  que  les  resultats  d’une  utilisation  concrete  n’auront  pas  ete  transposes 
dans  le  contexte  de  la  mise  en  oeuvre  des  capacites  produites.  Cependant,  les  produits  analytiques 
de  l’exercice  Gamma  ont  ete  bien  rccus  par  les  participants  et  par  les  commanditaires  qui 
representaient  des  organismes  cles  ayant  quelque  chose  a  voir  avec  l’eventuelle 
institutionnalisation  de  l’approche  DIGCap.  On  peut  done  considerer  que  l’exercice  Gamma  a 
marque  le  franchissement  d’une  etape  dans  la  progression  entre  le  stade  de  la  recherche - 
developpement  en  ingenierie  des  capacites  et  celui  de  1’ exploitation  de  celle-ci  par  les  milieux  du 
developpement  des  Forces. 
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1  Introduction 


Defence  R&D  Canada  (DRDC)  recently  originated  and  investigated  the  Capability  Engineering 
(CE)  concept  through  a  technology  demonstration  project  called  Collaborative  Capability 
Definition,  Engineering  and  Management  (CapDEM).  As  part  of  this  effort,  a  formal  evaluation 
strategy  for  the  approach  was  undertaken  [1],  The  strategy  involved  three  exercises  (Exercise 
Alpha,  Exercise  Beta  and  Exercise  Gamma),  each  one  applying  the  CapDEM  approach  with 
increasingly  larger  scale  and  complexity  (see  Section  2.1).  The  lessons  learned  were  then  used  to 
improve  and/or  clarify  the  elements  of  the  CapDEM  approach,  as  well  as  to  improve  the 
evaluation  methodology. 

The  scope  of  the  three  exercises  can  be  described  as  follows: 

•  Exercise  Alpha  [2]  [3]  [4]  was  the  initial  effort  to  evaluate  the  interaction  within  the 
CapDEM  approach.  Based  on  a  quasi-experimental  approach  and  using  a  semi-controlled 
environment1,  this  exercise  was  designed  to  be  a  ‘proof-of-concept’  trial  in  how  to  integrate 
and  evaluate  the  interdependencies  within  the  CapDEM  approach.  The  primary  goal  was  to 
‘debug’  the  evaluation  methodology  before  proceeding  with  larger  experimental  subject 
groups  in  subsequent  exercises. 

•  Exercise  Beta  [5]  [6]  [7]  was  intended  to  be  a  complete  functional  test  and  evaluation  of 
CapDEM  approach.  Based  on  a  realistic  problem  definition,  Exercise  Beta  shifted  emphasis 
from  debugging  the  evaluation  methodology  towards  the  core  of  the  problem  and 
CapDEM’s  ability  to  meet  client  needs. 

•  Exercise  Gamma  [8]  [9]  was  intended  to  be  a  complete  third-party  functional  test  and 
evaluation  of  CapDEM  approach.  Based  on  a  realistic  problem  definition,  Exercise  Gamma 
shifted  emphasis  towards  external  groups  being  able  to  address  their  own  problem  using  the 
proposed  approach. 

This  report  presents  the  results  of  Exercise  Gamma,  which  was  conducted  between  October  2006 
and  May  2007.  In  contrast  to  previous  exercises,  Exercise  Gamma  was  conducted  under  a  Project 
Level  Agreement  (PLA)  signed  by  three  DND  organizations,  the  purpose  of  which  was  to 
formally  engage  departmental  clients  (see  [10]  for  additional  perspective). 

The  remainder  of  this  report  is  organized  as  follows:  Chapter  2  briefly  describes  the  CapDEM 
approach,  the  previous  exercises  as  well  as  the  goals,  purpose  and  expected  outcomes  of  Exercise 
Gamma.  Chapter  3  then  outlines  the  exercise  results  in  terms  of  the  general  methodology,  while 
Chapter  4  presents  the  results  on  a  per  axis  basis.  Chapter  5  provides  discussion  and  analysis  of 
the  results  while  also  putting  forward  recommendations  in  support  of  eventual  institutionalization. 
Finally,  Chapter  6  concludes  with  a  summary  of  the  exercise  and  how  it  fulfilled  the  intent  of  the 
overall  evaluation  strategy. 


1  The  ‘quasi-experimental  approach’  refers  to  the  use  of  a  number  of  proven  experimental  procedures,  but 
not  to  the  extent  of  strictly  applying  the  scientific  method,  in  terms  of  a  separate  control  group  and  so  forth. 
Similarly,  the  ‘semi-controlled  environment’  refers  to  the  use  of  pre-defined  experimental  environment,  but 
one  that  was  allowed  to  evolve  to  best  support  the  effort  going  forward.  Note  that  all  exercises  within  the 
evaluation  strategy  were  addressed  in  this  way. 
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2  Background 


This  section  provides  a  brief  overview  of  the  CapDEM  approach,  summarizing  the  previous 
exercises  while  illustrating  the  intent  of  Exercise  Gamma. 

2.1  The  CapDEM  Approach 

In  terms  of  structure,  CapDEM  was  organized  along  three  primary  axes:  People,  Process  and 
Materiel.  Such  organization  provided  a  natural  division  for  issues  to  be  dealt  with  in  terms  of 
capabilities;  accordingly,  its  principle  output,  Capability  Engineering,  is  similarly  structured.  In 
short,  the  process  defines  the  rules  and  methodologies  by  which  the  people  apply  their  expertise, 
creativity  and  engineering  knowledge,  using  the  appropriate  technology  and  tools  to  facilitate  the 
development  of  the  necessary  capabilities.  Each  axis  can  be  defined  as  follows: 

•  People:  The  Capability  Engineering  Team  (CET)  construct  represents  a  central  and 
fundamental  element  of  organizational  design  for  CapDEM  (and  Capability  Engineering). 
The  CET  construct  [11]  is  defined  as  a  cross-functional,  multidisciplinary  team  (with 
complementary  skills)  committed  to  applying  and  managing  the  capability  engineering 
process.  The  CET  is  basically  composed  of  a  team  leader,  systems  engineers,  systems 
architects  and  requirements/operational  analysts.  This  core  analytical  team  is  then  partnered 
with  operational  subject  matter  experts  (SMEs)  and  liaison  members  from  across  the 
PRICIE2  capability  components. 

•  Process:  The  Capability  Engineering  Process  (CEP)  aims  at  providing  decision  makers 
with  better  information  on  strategic  investment  and  divestment  [12].  The  CEP  consists  of  a 
set  of  processes,  roles  and  responsibilities,  products  and  guidance  that  use  systems 
engineering  rigour  to  develop  potential  solutions  to  identified  capability  gaps.  These 
potential  solutions  (i.e.,  Force  Development  Options)  are  developed  using  the  five  inter¬ 
linked  processes  shown  as  a  system  of  gears  in  Figure  1.  This  figure  also  represents  the 
incremental  production  of  deliverables  as  through  the  continuous  and  dynamic  interaction 
between  the  processes.  As  changes  in  one  process  may  impact  another,  such  an  approach 
enables  pruning  of  the  solution  space  as  soon  as  possible  while  focusing  effort  on 
worthwhile  options  as  early  as  possible.  The  ‘Manage  Engineering  Effort’  process 
determines  the  speed  of  the  gears  with  regard  to  available  resources  and  constraint  (e.g.,  a 
delivery  date  given  by  the  decision  makers).  This  process  also  controls  the  decision  gates, 
the  passage  from  one  stage  to  the  next  (i.e.,  Inception,  Comprehension,  Elaboration  and 
Completion)  and  the  eventual  termination  of  the  whole  effort. 

•  Materiel:  The  Collaborative  Engineering  Environment  (CEE)  [13]  is  a  logical  environment 
consisting  of  a  set  of  tools  and  facilities  that  enable  information  exchange  and  collaboration 
among  engineers,  subject  matter  specialists  and  managers  at  multiple,  geographically 
distributed  locations  for  the  purpose  of  defining,  developing  and  evaluating  a  capability. 
Primarily  based  on  commercially  available  tools  and  applications  (i.e.,  COTS),  the  function 
of  such  an  environment  is  to  enable  project  stakeholders  to  have  a  common  location  and  a 


PRICIE  =  Personnel;  Research  and  Development;  Infrastructure  and  Organization;  Concepts,  Doctrine 
and  Collective  Training;  Information  Management;  Equipment,  Supplies  and  Services. 
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common  interface  by  which  to  access  information,  utilize  specialized  applications  and 
communicate/collaborate  with  each  other. 


Figure  1:  CEP  v4  Dynamics 


2.2  Exercise  Alpha 

Exercise  Alpha  [2]  [3]  [4]  was  conducted  in  spring  2005  with  an  abridged  CET  of  five  people. 
This  exercise  was  designed  to  be  a  ‘proof-of-concept’  trial  in  how  to  integrate  and  evaluate  the 
interdependencies  of  the  three  CapDEM  axes.  The  primary  goal  was  to  ‘debug’  the  evaluation 
methodology.  The  main  lessons  learned  related  to  the  areas  of  approach  and  methodology, 
(specifically  relating  to  scenario  selection),  CET  recruitment,  role  of  the  Evaluation  Team, 
suitability  of  data  collection  techniques  and  training. 

2.3  Exercise  Beta 

Exercise  Beta  [5] [6]  [7]  was  conducted  between  September  2005  and  February  2006.  This 
exercise  was  conducted  by  a  full  CET  consisting  of  DRDC  personnel  and  contractors  that  were 
geographically  distributed  between  DRDC  Ottawa  and  DRDC  Valcartier.  The  intent  was  to 
achieve  a  complete  functional  test  and  evaluation  of  the  three  CapDEM  axes.  Based  on  a  realistic 
problem  definition,  Exercise  Beta  shifted  emphasis  from  debugging  the  methodology  towards 
addressing  the  reality  of  the  problem  space  and  CapDEM’ s  ability  to  meet  client  needs.  The  main 
lessons  learned  from  this  exercise  were  in  terms  of  the  following  themes:  CET  organization, 
interaction  and  dedication;  assignment  of  operational  and  PRICIE  representatives;  CEP 
deliverable  orientation  and  tools;  CEE  awareness  and  IT  support;  Evaluation  Team  role  (coaching 
and  advisory);  and  training. 

2.4  Exercise  Gamma 

Exercise  Gamma  [8]  [9]  was  conducted  between  October  2006  and  May  2007.  The  intent  of  the 
exercise  was  to  execute  a  complete  third-party  functional  test  and  evaluation  of  CapDEM 
approach.  Based  on  a  realistic  problem  definition,  Exercise  Gamma  shifted  emphasis  towards 
external  groups  being  able  to  address  their  problem  using  the  proposed  approach.  As  part  of 
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addressing  a  client-based  problem  definition,  Exercise  Gamma  was  conducted  under  a  Project 
Level  Agreement  (PLA)  between  three  organizations:  Chief  Force  Development  (CFD),  Chief  of 
Staff  Information  Management  (COS(IM)  /  ADM(IM))  and  Defence  R&D  Canada  (DRDC)  [10]. 
This  approach  provided  for  the  formal  engagement  of  departmental  clients  and  access  to 
department  staff  for  purposes  of  forming  the  CET.  While  such  an  approach  was  different  from 
the  initial  design  of  Exercise  Gamma  [1],  the  objectives  between  the  PLA  and  the  original 
evaluation  strategy  were  compatible.  Table  2  presents  a  description  of  Exercise  Gamma, 
summarizing  goals,  scope  and  expected  outputs  for  each  axis  and  the  overall  exercise. 

Table  1:  Exercise  Gamma  -  Design  Summary 


Aspect  Description 


Goals 

Overall 

•  Test  and  evaluate  the  CET,  CEP  and  CEE  as  applied  to  a  realistic  problem 
provided  by  an  external  client. 

•  Ensure  the  necessary  people,  process  and  materiel  can  address  a  ‘real 
world’  problem  when  being  executed  by  an  appropriate  real-world  client 
group. 

People 

•  Test  and  evaluate  the  final  iteration  of  the  CET. 

•  Put  forward  best  practices  and  lessons  learned  for  a  comprehensive  Team 
Charter,  including  the  proper  identification  of  roles  and  responsibilities, 
dynamic  teamwork  and  collaboration  practices,  as  well  as  effective  team 
communication  practices  and  mechanisms. 

Process 

•  Test  and  evaluate  version  4  of  the  CEP,  including  its  complete  set  of 
activities  and  deliverables. 

Materiel 

•  Test  and  evaluate  the  revised  CEE. 

•  Investigate  and  determine  a  reasonable  list  of  functionalities  (required, 
preferred  and  optional)  for  CEE  institutionalization. 

Scope 

Overall 

•  Seven  month  duration. 

•  ‘Real  world’  scenario. 

People 

•  Complete  team  size. 

•  DND  clients  from  more  than  one  organization. 

Process 

•  Complete  each  CEP  v4  deliverable. 

•  Complete  each  CEP  v4  task  and  iterate  as  mandated  by  the  process. 

Materiel 

•  Fully  exploit  CEE  functionalities  in  performance  of  their  tasks. 
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People 

•  Complete  the  Team  Charter,  incorporating  roles  and  responsibilities,  as 
well  as  teamwork  and  collaboration  principles  for  the  CET. 

•  Put  forward  recommended  best  practices  for  initiating  a  successful  CET. 

•  Link  successful  team  dynamics  with  CEE  (e.g.,  ACCESS  Labs  and 
Livelink)  for  enhanced  communication  practices. 

•  Ensure  the  CET  clearly  understands  CEP  processes. 

c n 

£ 

•  Ensure  the  CET  clearly  understands  CEP  deliverables. 

O 

u 

•  Identify  documentation  weaknesses  (process  and  deliverable  templates)  to 

o 

facilitate  application  during  institutionalization  within  the  DND/CF. 

'O 

a 

ct 

Process 

•  Identify  training  weaknesses  to  be  addressed  when  providing  future 

c n 

S3 

training  to  DND/CF  personnel. 

a 

•  Identify  metrics  to  measure  and  facilitate  continuous  improvement  of  the 

o 

CEP. 

•  Identify  critical  factors  for  successful  implementation  of  the  CEP. 

•  Provide  a  user-validated  list  of  mandatory/preferred  functionalities  to  fully 

enable  the  CEP  and  facilitate  collaboration  inside  the  CET. 

Materiel 

•  Refine  and  validate  the  existing  workflow  between  the  various  tools. 

•  Provide  an  updated  and  expanded  database  of  ‘Frequently  Asked 

Questions’  (FAQ),  ‘Tips  &  Tricks’  and  lessons  learned. 

DRDC  Ottawa  TR  2011-044 


5 


3  Approach  and  Methodology 


Given  the  introduction  to  the  CapDEM  evaluation  strategy,  this  section  now  overviews  the 
approach  and  methodology  applied  during  Exercise  Gamma.  As  planned,  a  large  number  of  the 
lessons  learned  from  previous  exercises  (in  particular,  Exercise  Beta)  were  applied  towards  the 
realization  of  Exercise  Gamma;  these  include:  modifying  CET  organization,  adjusting  and 
extending  the  training  programme;  employing  an  improved  version  of  the  CEP,  and  having  the 
Evaluation  Team  attempt  to  be  more  proactive. 

3.1  Data  Gathering  and  Analysis 

The  approach  to  data  analysis  within  Exercise  Gamma  remained  almost  identical  to  that 
employed  in  Beta  Exercise,  primarily  focusing  on  common  themes  and  issues.  Table  2 
summarizes  the  means  and  mechanisms  used  for  data  collection.  Each  of  the  means/mechanisms 
used  had  specific  purposes  while  also  contributing  to  the  development  of  the  focus  group 
questionnaires.  Observations,  CET  deliverables,  issues  and  concerns  as  well  as  requests  for  help 
and  coaching  provided  good  sources  of  information  from  which  to  build  more  focused  questions. 
As  with  Exercise  Beta,  the  focus  group  became  the  principle  and  best  source  of  data,  as  most  of 
the  information  gathered  through  other  means  were  validated  with  all  the  CET  members  during 
the  focus  group  sessions.  Hence,  the  themes  and  issues  that  are  presented  in  the  following 
sections  of  this  report  primarily  stem  from  the  focus  group  analysis. 


Table  2:  Data  Collection  Means  and  Mechanisms 


Means 

Purpose 

Comment 

CET  Meeting 
Attendance 

•  Observe  and  collect  information 
on  the  team’s  progress. 

•  Suggest  approaches  to  issues. 

•  Answer  questions. 

One  or  more  members  of  the  Evaluation 
Team  attended  most  meetings. 

Individual 

Meetings 

•  Discuss  a  participant’s  concerns 
on  an  individual  basis. 

•  Provide  advice  to  a  participant 
on  an  individual  basis. 

On  an  ad  hoc  basis,  as  requested  by  a 
CET  or  an  Evaluation  Team  member. 

Gate  Review 
Meeting 

•  Evaluate  the  content  of  the  work 
produced  at  each  stage. 

Performed  at  the  end  of  Inception  and 
Comprehension  stages. 

Examine  CET 
Deliverables 

•  Evaluate  the  content  of  the  work 
produced  at  each  stage. 

Performed  at  the  end  of  Inception, 
Comprehension  and  Completion  stages. 
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Focus  Group 

•  Extract  common  themes  and 
issues  related  to  each  axis  as 
well  as  the  methodology  and 
approach  relative  to  each  stage. 

Performed  at  the  end  of  Inception, 
Comprehension  and  Completion  stages; 
also  performed  midway  through  the 
Comprehension  stage. 

Coaching  and 
Help  Requests 

•  Support  to  ensure  the  exercise 
would  run  smoothly. 

•  Help  identify  any  weaknesses  in 
the  CapDEM  approach. 

Adaptation  of  the  ‘Help  Desk’  concept 
from  Exercise  Beta;  the  coaching 
practices  evolved  over  the  duration  of 
the  exercise. 

3.2  Work  Plan 

The  exercise  work  plan  [8]  organized  the  entire  effort  into  three  main  phases: 

•  Preparation:  This  phase  consisted  of  ‘pre-CET’  activities  to  ensure  participant  readiness  as 
well  as  to  address  any  issues  related  to  the  scenario,  training,  evaluation  or  technical 
environment. 

•  Experimentation:  This  phase  corresponded  to  the  period  in  which  the  CET  applied  the 
CEP  using  the  CEE.  Thus,  this  phase  aligned  to  the  actual  execution  of  the  CapDEM 
approach. 

•  Analysis:  This  phase  corresponded  to  the  organization,  examination  and  analysis  of  the 
feedback  collected  (i.e.,  the  ‘post-CET’  phase). 

Unlike  previous  exercises,  the  Evaluation  Team  did  not  strictly  control  the  schedule  for  Exercise 
Gamma.  Rather,  the  Evaluation  Team  proposed  a  timeline  for  CEP  execution  (i.e.,  dates  for  the 
various  stages  and  iterations)  in  order  to  help  meet  the  schedule  (i.e.,  phases  and  deadlines) 
outlined  in  the  PLA.  The  exercise’s  final  timeline  (i.e.,  that  actually  realized)  is  presented  in 
Figure  2,  illustrating  both  exercise  phases  and  CEP  stages.  The  dark  bars  correspond  to 
Evaluation  Team  involvement  while  the  light  grey  bars  correspond  to  CET  involvement. 

As  planned,  the  Experimentation  Phase  of  Exercise  Gamma  started  on  25  September  2006  and 
ended  mid-May  2007,  approximately  2-3  weeks  later  than  anticipated.  The  main  differences  from 
the  proposed  timeline  were: 


Gamma  Experimentation  Cycle 

Preparation  Phase 

28  Aug  06  to  6  Oct  06 

Experimentation  Phase  |||||||||||||  |||||||||||||||||||||  25  Sept  06  to  25  Apr  07 

Inception  Stage 

25  Sept  06  to  27  Oct  06 

Comprehension  Stage 

30  Oct  06  to  6  Apr  06 

Elaboration  Stage 

9  Apr  07  to  1 1  May  07 

Completion  Stage 

1 4  May  07  to  25  May  07 

Analysis  Phase 

||  Between  July  and  August 

Figure  2:  Gamma  Experimentation  Cycle 
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•  The  Comprehension  Stage  was  much  longer  than  expected.  This  stage  lasted  2 1  weeks  with 
only  one  iteration  while  the  Evaluation  Team  had  suggested  8  weeks  with  2  iterations. 

•  The  Elaboration  Stage  was  much  shorter  than  expected.  This  stage  lasted  5  weeks  with  only 
one  iteration  while  the  Evaluation  Team  had  suggested  13  weeks  with  3  iterations. 

Since  the  PLA  governing  the  exercise  specified  a  firm  schedule,  there  was  no  ability  to 
significantly  extend  the  end  date;  hence,  the  extra  time  consumed  during  the  Comprehension 
Stage  came  at  the  expense  of  the  Elaboration  Stage. 

3.3  Capability  Engineering  Team 

Exercise  Gamma’s  initial  organizational  chart,  as  presented  in  the  work  plan  and  PLA,  was  never 
completely  realized  for  the  exercise.  In  spite  of  significant  effort  to  obtain  sufficient  staffing,  the 
roles  of  CET  Coordinator,  PRICIE  and  ECS  Operational  SMEs,  as  well  as  Industry  Interface 
were  not  filled  (Figure  3).  A  contractor  did  partially  assume  some  of  the  tasks  intended  for  the 
CET  Coordinator,  but  only  for  a  three  month  period.  Additionally,  the  composition  of  the  CET 
changed  over  the  course  of  the  exercise  due  to  the  retirement  of  the  initial  CET  Lead.  Subsequent 
changes  included  selecting  the  original  Chief  Operational  Architect  as  the  new  CET  Lead  along 
with  the  use  of  a  contractor  to  backfill  the  subsequently  empty  Chief  Operational  Architecture 
role.  The  final  resulting  CET  organizational  chart  is  shown  in  Figure  4. 


Figure  3:  Intended  Gamma  Structure 
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Figure  4:  Final  Gamma  CET 


3.4  Evaluation  Team 

The  Evaluation  Team  was  initially  composed  of  five  DRDC  scientific  and  technical  staff 
geographically  distributed  between  Ottawa  and  Valcartier,  each  with  a  particular  expertise 
relative  to  the  CapDEM  approach.  Unlike  most  members  of  the  CET,  the  majority  of  the 
Evaluation  Team  had  experience  with  the  previous  CapDEM  evaluation  exercises,  Exercise 
Alpha  and  Exercise  Beta. 

For  Exercise  Gamma,  the  main  responsibilities  for  the  Evaluation  Team  were  as  follows: 

•  Oversee  exercise  progress 

•  Organize  CET  training 

•  Develop  tools  and  mechanisms  to  gather  data  for  each  axis 

•  Provide  coaching  to  the  CET  regarding  issues  related  to  each  axis 

•  Provide  guidance  to  organize  and  plan  CET  activities 

•  Implement  and  follow-up  on  evaluation  activities 

•  Report  results 

Unfortunately,  due  to  a  variety  of  circumstances,  the  Evaluation  Team  had  to  work  with  only  four 
members  for  the  majority  of  the  exercise.  Further,  at  some  points  there  were  only  three 
Evaluation  Team  members  available,  thus  leaving  some  axes  without  an  advisor. 

3.5  Scenario 

For  Exercise  Gamma,  scenario  selection  was  performed  by  the  exercise  sponsor(s)  as  opposed  to 
the  Evaluation  Team,  as  in  previous  CapDEM  evaluation  exercises.  As  such,  it  was  chosen  to 
meet  a  specific  objective  of  the  PLA:  “to  define  C4+I3  capability  gaps  and  recommend  Force 
Development  options”. 
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The  ‘I’  in  C4+I  represents  ‘Information’  (versus  C4I,  in  which  ‘I’  stands  for  ‘Intelligence’). 
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The  chosen  capability  domain  of  Command,  Control,  Communication,  Computers  and 
Information  (i.e.,  C4+I)  is  a  sub-element  of  the  broader  C4ISR  domain.  As  the  C4+I  capability  is 
applicable  across  many  scenarios,  the  problem  had  to  be  further  scoped.  Hence,  it  was  decided 
that  the  CET  would  develop  C4+I  options  to  contribute  to  a  domestic  humanitarian  disaster 
response  and  relief  effort  as  led  by  Public  Safety  and  Emergency  Preparedness  Canada  (PSEPC) 
in  a  2015  timeframe. 

3.6  Training 

The  initial  Exercise  Gamma  work  plan  proposed  a  number  of  mandatory  and  optional  training 
sessions  to  help  the  CET  in  their  application  of  the  CapDEM  approach.  A  list  of  the  mandatory 
training  components  given  is  listed  in  Table  3.  However,  three  courses  had  to  be  cancelled: 

•  CEE  Overview  and  System  Engineering  Tool  Synopsis:  The  one-day  training  session 
which  was  to  focus  on  Materiel  axis  considerations  had  to  be  cancelled  due  to  the  difficulty 


Table  3:  Training  Summary 


Date 

Training 

Description 

Duration 

14-15 

September 

2006 

Architecture 

Framework 

Orientation 

Familiarization  on  architectures  and  their 
application  (including  some  hands-on 
practice) 

2  days 

18-20 

September 

2006 

Livelink 

User 

Training 

Introduction  covering  the  various 
capabilities  and  day-to-day  use  of  Livelink 
from  the  end-user  perspective 

21/2  days 

27-28 

September 

2006 

CEP  v4 

Parts  1-2 

Overview  of  the  CEP  and  details  on  the 
Inception  Stage 

2  days 

2 

October 

2006 

Use  Case 
Overview; 
Team 
Dynamics 

Overview  of  use  case,  project  charter  and 
human  dynamics  (including  an  exercise  to 
illustrate  the  value  of  team  collaboration 
and  communication) 

Vi  day 

30 

October 

2006 

CEP  v4 

Part  3 

Details  on  the  Comprehension  Stage 

1  day 

15-16 

February 

2007 

CEP  v4 

Part  4 

Details  on  the  Elaboration  Stage 
(including  hands-on  component) 

2  days 
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in  defining  the  specific  CEE  for  Exercise  Gamma,  since  the  CORA  ACCESS  Lab  and  the 
associated  system  engineering  tools  were  in  the  process  of  obtaining  secret  certification. 

•  Overview/Tutorial  on  Phoenix  Integration’s  Model  Center:  The  half-day  course  to 
overview  this  specific  tool  used  for  model  analysis  and  optimization  purposes  was  not  held, 
due  to  a  combination  of  logistical  and  resource  issues.  A  familiarization  briefing  was 
substituted  later  in  the  exercise  to  help  set  the  context  for  its  use. 

•  CEP  v4  (Part  5):  This  portion  of  the  CEP  training  (Completion  Stage)  was  omitted  due  to 
time  constraints  and  the  relative  importance  of  this  stage  vs.  others  in  meeting  the  required 
objectives. 
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4  Results 


This  section  presents  the  results  from  Exercise  Gamma,  organized  relative  to  the  three  axes  as 
well  as  the  approach  and  methodology  taken  within  the  exercise.  Results  from  applying  the 
CapDEM  approach  in  terms  of  CET  output  are  also  provided.  Analysis,  observations  and  lessons 
learned  relative  to  these  results  then  follow  in  Section  5.  Note  that  all  quotations  within  this 
document  come  from  the  CET,  unless  otherwise  stated. 

4.1  Approach  and  Methodology 

The  following  section  provides  an  overview  of  results  relative  to  the  whole  of  Exercise  Gamma 
and  its  methodology.  These  include  the  areas  of  data  gathering,  participation,  questioning 
structure  and  thematic  analysis.  Training  and  Evaluation  Team-related  issues  are  also 
highlighted. 

4.1.1  Data  Gathering 

Exercise  Gamma  employed  a  data  collection  methodology  based  on  the  use  of  focus  groups, 
similar  to  that  used  in  Exercise  Beta.  This  approach  allowed  participants  to  put  forward  their 
ideas  and  issues  within  a  forum  setting.  The  focus  groups  were  scheduled  to  coincide  with  CEP 
gate  reviews  in  order  to  maximize  attendance  and  participation  by  the  CET,  while  also  aligning 
the  data  collection  points  to  well  established  ‘synchronization  points’  in  the  process.  However, 
due  to  unplanned  irregularities  in  exercise  execution,  changes  were  required  to  take  into  account 
the  departure  of  the  initial  CET  Lead  as  well  as  schedule  delays  encountered  by  CET  (see 
Section  3.2). 

4.1.2  Participation 

The  Evaluation  Team  conducted  four  focus  groups  between  November  2006  and  May  2007  as 
shown  in  Table  4.  The  focus  groups  obtained  substantial  levels  of  participation  and  the  number 
of  attendees  for  each  focus  group  was  almost  constant;  however,  the  specific  participants  were  not 
always  the  same.  For  example,  only  three  CET  members  attended  all  focus  groups,  while  three 
members  each  missed  one  focus  group  and  two  others  only  attended  two  focus  groups  (as  a  result 
of  changing  CET  membership).  Further,  one  Evaluation  Team  member  was  missing  from  each 
session  (not  the  same  each  time)  while  the  observers  also  changed  over  time.  CET  members  who 
were  unable  to  attend  were  allowed  to  submit  their  commentary  (in  advance). 

4.1.3  Question  Structure 

A  semi-structured  interview  guide  was  employed  throughout  all  of  the  focus  groups  (see  Annex  A 
through  Annex  D).  Approximately  14  to  17  questions  were  asked  per  focus  group,  concentrating 
mainly  on  the  people,  process  and  materiel  axes,  training  and  Evaluation  Team  support.  The 
duration  of  each  focus  group  was  limited  to  no  more  than  three  hours  to  ensure  enough  time  for 
the  questions  to  be  properly  addressed  while  still  keeping  CET  attention.  The  questions  were 
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Table  4:  Focus  Group  Summary 


Corresponding 

Attendees 

Event 

uaic  E/utaiiuii 

CET 

Evaluation 

Team 

Other 

End  of 

Inception  Stage 

1  November 
2006 

CORA 

ACCESS  Lab 

7 

4 

2 

Change  of  CET  Lead 

20  December 
2006 

CORA 

ACCESS  Lab 

6 

4 

2 

End  of 

Comprehension 

Stage 

1 1  April 

2007 

NDHQ 

Conference  Room 

6 

4 

2 

End  of  Exercise 
(After  both 

Elaboration  and 
Completion  Stages) 

25  May 

2007 

NDHQ 

Conference  Room 

6 

4 

2 

intended  to  start  conversations  about  general  areas  and  themes  that  were  familiar  to  the 
participants.  The  questions  were  generally  easy  to  understand  and  were  predominantly  open- 
ended.  A  trained  facilitator  moderated  the  focus  group  sessions  and  used  additional  probing 
questions  to  obtain  a  better  understanding  of  the  responses  to  the  key  questions.  Prior  to  running 
the  focus  groups,  the  Evaluation  Team  vetted  the  questions  in  order  to  ensure  suitability  for  use 
with  the  CET.  Informed  by  previous  experience  in  Exercise  Beta,  question  development  was 
based  on  the  work  the  CET  had  to  perform  during  the  specific  period,  the  observations  made  by 
the  Evaluation  Team  during  CET  work  sessions,  plus  any  feedback  or  questions  received  from  the 
CET  during  the  same  period. 


4.1.4  Thematic  Analysis 

The  results  for  Exercise  Gamma  have  been  organized  based  on  an  analysis  of  common  themes 
which  emerged  throughout  the  various  stages  of  the  exercise.  Unless  there  was  something 
specific  to  a  period,  the  results  are  presented  as  a  roll-up  of  the  critical  areas  that  were  identified 
for  each  axis.  Prior  to  the  analysis  proper,  the  notes  taken  by  the  Evaluation  Team  members  were 
integrated  together,  reinforced  by  audio  recordings  of  the  focus  group  sessions  should 
clarification  be  required.  Subsequently,  mind  mapping  software  was  used  to  assist  in  the  analysis 
of  the  consolidated  notes  and  in  drawing  a  spatial  representation  of  the  main  themes  and  their 
relationships  that  emerged  from  each  focus  group  (see  Annex  A  through  Annex  D). 
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4.1.5  Training 


The  general  feedback  regarding  training  identified  the  need  for  more  highly  focused  training,  to 
the  degree  of  being  context  specific  and  directly  applicable  to  the  specific  tasks  the  individuals 
had  to  perform.  The  degree  of  participatory  (i.e.,  ‘hands-on’)  training  was  deemed  insufficient  at 
the  beginning,  but  reported  as  more  satisfactory  later  in  the  exercise.  Furthermore,  it  was  noted 
that  training  needed  to  be  conducted  by  those  who  best  know  how  to  train  (i.e.,  genuine 
instructors),  rather  than  rely  on  ad  hoc  personnel.  The  issues  of  consistency  across  different 
trainers  and  timeliness  in  terms  of  delivery  were  also  highlighted.  Other  concerns  raised  were: 
(1)  the  balance  of  individual  versus  group  training;  (2)  the  need  for  training  on  specialized  tools 
in  context  with  specific  roles;  and  (3)  the  role  of  coaching  and  how  it  was  provided. 

In  terms  of  per  axis  training,  the  Process  and  Materiel  axes  garnered  the  most  feedback. 
Specifically,  process  training  was  not  adequately  participatory  and  there  was  the  desire  to  have 
the  CET  trained  with  consideration  of  (i.e.,  linkage  to)  the  ‘big  picture’  topic  as  well  as  the 
capability  gap  itself.  Furthermore,  the  workflow  was  also  not  ready  at  the  appropriate  time. 

In  terms  of  the  Materiel  axis,  the  amount  of  training  completed  was  viewed  as  insufficient 
(approximately  50%  of  the  intended  curriculum  was  actually  given4).  There  was  also  concern 
over  the  alignment  of  the  training;  for  example,  the  Department  of  Defense  Architecture 
Framework  (DoDAF)  training  was  not  targeted  towards  Exercise  Gamma  but  attempted  to 
leverage  training  that  was  arranged  as  part  of  the  Marine  Security  Operations  Centre  (MSOC) 
project. 

4.1.6  Evaluation  Team 

The  general  feedback  in  terms  of  the  Evaluation  Team  addressed  two  main  areas:  (1)  their  role 
and  level  of  support  with  respect  to  the  CET;  and  (2)  their  involvement  in  training  and  coaching. 

In  terms  of  their  role  and  level  of  support,  the  Evaluation  Team  was  quoted  as  being  “very 
responsive”  to  the  needs  of  the  CET.  However,  it  was  noted  that  the  use  of  facilitators  within 
meetings  would  have  been  useful  and  that  the  CET  did  not  like  a  ‘style’  of  support/feedback  in 
which  the  Evaluation  Team  would  ‘ask  questions’  instead  of  providing  immediate  answers. 
Furthermore,  the  CET  expressed  discomfort  at  the  use  of  the  term  ‘Evaluation  Team’,  despite  the 
rationale  and  explanation  provided. 

In  terms  of  involvement  in  the  training  and  coaching,  there  was  a  strong  desire  for  coaching  to  be 
provided  in  addition  to  the  training  sessions.  Feedback  stated  that  coaching  was  critical  and  had 
definite  benefits  (“it’s  a  requirement”).  However,  the  coaching  provided  was  not  viewed  as 
sufficient,  appropriate  or  what  was  expected.  Specifically,  the  Evaluation  Team  was  not  seen  as 
sufficiently  proactive  in  terms  of  the  requested  level  of  coaching  (“...  frame  the  meeting,  be 
actively  involved,  come  back  after  meeting  and  give  immediate  feedback”).  Furthermore,  it  was 
noted  that  coaching  needs  to  be  performed  by  experienced  people  and  that  embedding  a  ‘Process 
person’  within  the  CET  “should  have  been  an  option”.  Finally,  it  was  stated  that  coaching  is 
required  across  all  axes,  not  just  in  terms  of  the  CEP. 


4  Refer  to  Section  3.6  for  further  information. 
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4.2  CapDEM  Axes 


The  following  section  now  outlines  the  results  obtained  from  Exercise  Gamma  with  respect  to 
each  axis.  An  overview  mind  map  can  be  found  at  the  end  of  the  respective  sections  (Figure  5  for 
People,  Figure  6  for  Process  and  Figure  7  for  Materiel). 

4.2.1  People 

Challenges  within  the  People  axis  were  primarily  related  to  collaboration  and  communication,  the 
team  charter,  leadership,  roles  and  responsibilities,  and  use  of  contractors. 

Collaboration  and  communication.  As  the  effort  started,  progress  relied  on  the  availability  of 
documents  as  part  of  the  team  ‘finding  their  way’.  However,  as  the  team  increasingly  understood 
what  they  were  supposed  to  do,  a  suitable  collaborative  protocol  evolved.  Thus,  team 
collaboration  was  generally  effective  but  with  a  great  reliance  on  face-to-face  communication. 
While  the  CET  did  feel  that  they  communicated  well,  there  was  evidence  of  fragmented  internal 
communication  within  the  CET;  for  example,  meetings  were  (at  one  point  in  time)  regarded  as 
tedious  with  a  lack  of  discipline.  The  acknowledgment  of  their  collaborative  challenges 
coincided  with  the  arrival  of  a  new  CET  Lead,  who  changed  the  way  the  team  worked.  At  that 
time,  they  self-organized  in  small  focussed  groups  to  make  working  easier;  however,  such  an 
approach  underlined  the  need  to  be  more  aware  of  information  exchange  between  those  groups: 
“[The]  team  leader  encouraged  small  focus  subgroups  around  specific  themes — that  worked 
well — it  allowed  concurrent  issue  working  but  the  downside  [was  the]  required  communication 
and  what  [was]  going  on”.  Relatedly,  some  CET  members  were  concerned  about  commenting  on 
others’  responsibilities/products.  As  one  person  stated:  “I  don’t  often  know  what  other  people 
are  doing,  why  they  are  doing  it  and  what  it  means  to  me...  [I]  don’t  want  to  comment/have  the 
capability  or  authority  to  comment  because  of  this”.  Notably,  the  CET  felt  that  they  were  better 
able  to  fulfill  their  responsibilities  and  felt  more  productive  when  working  within  sub-teams. 

Team  charter.  Although  the  team  charter  was  introduced  as  part  of  the  initial  training,  the  CET 
did  not  continue  to  actively  utilize  it  and  examine  how  it  could  be  used  to  support  team  effort 
throughout  the  exercise  (e.g.,  no  team  discussion  on  their  own  ‘rules  of  engagement’).  The  CET 
felt  that  there  was  inconsistency  in  role  definition  between  PLA,  CEP  and  team  charter  (e.g.,  “Not 
doing  what  is  defined  in  the  charter”;  “I  am  doing  more  than  what  I  am  supposed  to  do”). 
Furthermore,  a  lack  of  strong  coordination  and  varied  priorities  between  the  team  charter  and 
other  governing  documents  (such  as  the  PLA)  directly  affected  exercise  execution  (i.e.,  the  CET 
Lead  stated  that  “[the]  true  deadlines  were  the  PLA  ones”). 

Leadership.  Leadership  impacted  the  CET  in  terms  of  knowledge,  style  and  approach. 
Specifically,  the  background  of  CET  Lead  was  deemed  important  (“It’s  important  that  the  leader 
is  knowledgeable  to  the  domain”).  Furthermore,  there  is  the  need  for  process  expertise  within  the 
team  leadership  such  that  the  team  leader  requires  a  global  vision  of  the  process  in  order  to  be 
effective.  There  was  also  a  marked  change  in  the  style  of/approach  to  leadership  employed  over 
the  duration  of  the  exercise.  While  the  Evaluation  Team  was  consciously  non-obtrusive  to 
facilitate  an  independent  CET,  for  a  significant  portion  of  the  exercise,  select  contractor(s) 
frequently  interrupted  CET  proceedings,  causing  confusion  and  undermining  leadership.  Once 
this  situation  was  addressed,  fewer  interruptions  and  unexpected  inputs  resulted  in  improved 
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leadership  focus.  Leadership  focus,  however,  was  also  impacted  by  the  CET  Lead  performing 
many  of  the  functions  assigned  to  other  roles  (e.g.,  CET  Coordinator)  that  were  not  filled  (see 
section  on  Roles  and  Responsibilities).  Consequently,  leadership  was  considered  crucial  and  as 
was  ensuring  the  right  people  were  in  the  right  roles  (e.g.,  “If  [we  had]  lesser  leadership,  we  could 
have  really  been  in  trouble”). 

Roles,  responsibilities  and  the  use  of  contractors.  The  clarity  of  roles,  what  the  associated 
responsibilities  were  and  who  was  filling  those  positions  was  not  always  clear  throughout  the 
exercise.  In  response  to  mismatches  in  terms  of  work  styles  and  expectations  (“some  work  not 
being  done”),  select  individuals  took  it  upon  themselves  to  perform  work  that  was  the 
responsibility  of  other  CET  members5.  Meanwhile,  some  CET  members  felt  uncomfortable  with 
contractors  being  too  closely  associated  with  the  CET  (e.g.,  attending  all  meetings),  such  that 
some  contractors  unduly  influenced  the  work  and  the  way  the  job  was  done  (“We  may  be 
spending  more  time  discussing  tasks  with  contractors”).  Those  contractors  who  had  focused  tasks 
were  appreciated  while  the  CET  did  acknowledge  their  accountability  on  this  issue  (“The  value  of 
contractors  is  to  provide  them  specific  instructions  with  very  clear  guidelines,  when  done  and 
how  to  do  it  -  we  failed  to  do  that”).  The  CET  did  not  feel  that  they  were  in  control,  or  that  they 
had  the  authority  to  change  organization  of  the  team.  Indeed,  the  team  leader  did  not  always 
know  the  purpose  of  certain  individuals  and  why  there  were  at  a  meeting.  The  contractors 
sometimes  seemed  to  be  more  confident  than  the  CET  on  the  work  to  be  done,  thus  leading  to  an 
awkward  dynamic  on  the  team. 

Team  dynamics  were  also  affected  by  the  perceived  optics  of  the  exercise’s  organizational  chart. 
Specifically,  some  individuals  fixated  on  the  title  of  a  role  and  the  position  on  the  chart  as 
implying  a  support  position  (e.g.,  the  feeling  of  being  ‘demoted’).  This  approach  also  affected 
how  certain  roles  were  regarded,  such  that  the  CET  rationalized  that  the  role  CET  Coordinator 
was  primarily  administrative.  Conversely,  the  CET  Lead  did  not  feel  he  had  the 
authority/mandate  to  engage  with  SMEs.  The  result  was  an  incomplete  CET  that  increasingly 
realized  the  importance  of  these  positions  over  the  course  of  the  exercise. 

In  line  with  the  issues  mentioned  above  under  ‘Team  Charter’,  there  was  also  confusion  in  terms 
of  roles  and  responsibilities  due  to  the  multiplicity  of  events  and  initiatives  surrounding  the 
execution  of  Exercise  Gamma.  Specifically,  the  confluence  of  the  PLA,  the  exercise  workplan, 
and  input  from  the  Capability  Management  Working  Group  (CMWG)  resulted  in  different 
direction  coming  from  different  leaders.  Indeed,  various  individuals  were  aware  of/involved  in 
more  than  one  effort,  which  created  a  tension  in  terms  of  execution  (“They  were  more 
emotionally  involved  in  the  PLA  than  in  the  CET  initiative”).  Conversely,  the  CET  felt  there  was 
a  need  for  a  better  interface  with  Capability-Based  Planning  (CBP)  and  its  planners  (e.g.,  “We 
need  someone  [from]  CBP  to  be  more  CEP  aware  to  do  what  we  tried  to  do”). 

Other  issues  included  personnel  availability  as  well  as  timeline  and  pacing.  Specifically,  the  CET 
felt  that  there  was  not  enough  time  to  complete  the  work  within  the  Comprehension  Stage 
(i.e.,  sense  of  urgency  to  complete  this  stage).  For  example,  one  person  stated:  “We  saw  the  light 
just  in  April  to  understand  what  [it]  was  that  we  needed  to  know.  We  missed  one  or  two  months”. 


5  Conversely,  various  members  wanted  someone  to  operate  the  tools  on  their  behalf,  particularly  in  terms 
information  management  (e.g.,  Livelink)  and  the  general  organization/configuration  management  of  the 
CEE.  This  issue  is  further  explored  under  the  Materiel  axis  (Sections  4.2.3  and  5.1.3). 
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This  issue  was  exacerbated  by  concerns  over  CET  member  availability  (e.g.,  full-time 
commitment  to  the  effort)  combined  with  the  sentiment  that  there  was  too  much  emphasis  on 
project  management  (i.e.,  more  time  should  have  been  spent  on  completing  tasks  than  on  project 
management).  One  person  stated:  “I  feel  much  time  was  lost  to  scheduling  when  we  could  have 
been  completing  our  individual  tasks”. 
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Figure  5:  Summary  Map  -  People 


4.2.2  Process 

Challenges  within  the  Process  axis  were  primarily  related  to  cultural  resistance  and  the 
comprehension  of  how  the  numerous  elements  of  the  exercise  would  coalesce  together.  Within 
this  section,  results  are  summarized  along  the  following  themes:  understanding  and  assessment, 
documentation,  process  constructs,  capability  gap,  deliverables,  knowledge  and  timeline.  As 
discussed  further  in  the  analysis  section  (Section  5.1.2),  various  improvements  to  the  CEP  were 
realized  as  a  result  of  these  challenges,  some  of  which  were  applied  within  the  exercise  itself. 
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4.2.2. 1  Understanding  and  Assessment 


Challenges  in  terms  of  understanding  and  assessment  of  the  CEP  appear  to  have  stemmed 
primarily  from  a  mix  of  organizational  cultures  and  linkage  to  a  broader  departmental  context 
along  with  timely  understanding  and  its  iterative  nature. 

Culture  and  context.  There  was  diverse  perception  in  the  level  of  achievement  across  the  CET. 
Specifically,  there  was  a  marked  difference  in  perception  with  respect  to  the  degree  of  task 
completion  and  there  was  also  an  issue  of  resistance  to  unfamiliar  work  practices  (such  as 
architectures).  Such  variation  corresponded  to  differences  in  team  demographics  (e.g.,  military 
vs.  department  vs.  scientist/engineer),  as  did  expectations  in  terms  of  outputs  from  the  exercise. 
For  example,  there  was  an  initial  lack  of  understanding  over  what  outputs  were  to  be  obtained  (at 
the  Inception  Stage).  Specifically,  there  was  contention  over  the  deliverables  (and  artefacts)  as 
specified  by  the  CEP  versus  those  outlined  in  the  PLA.  For  example,  concern  was  expressed 
whether  some  of  the  outputs  would  be  “what  the  VCDS  wants  to  see”,  despite  being  at  an  early 
stage  in  the  effort.  Similarly,  there  was  also  initial  uncertainty  over  the  intent  of  the  gate 
review(s);  that  is,  the  structure,  organization  and  purpose  of  the  gate  review  (particularly  for  the 
Inception  Stage)  were  not  clear  a  priori.  Specifically,  the  CET  did  not  know  who  would  approve 
their  work  or  what  would  be  the  conditions  needed  to  meet  expectations.  For  example,  some 
participants  errantly  thought  that  the  Evaluation  Team  was  the  approval  authority  at  the  gate 
review  rather  than  the  Capability  Manager  (e.g.,  “Not  sure  who  was  the  approval  authority”;  “I 
didn’t  know  who  was  going  to  give  ‘nay’  or  ‘yea’”;  “Not  sure  of  the  passing  criteria”;  “We  tried 
to  guess  at  criteria  at  passing  the  gate. . .  there  was  none  to  refer  to”).  Subject  to  correction  of  the 
gate  review  presentation  template  combined  with  an  explanation  of  intent  by  the  Evaluation 
Team,  this  issue  was  not  raised  again. 

Linkage  to  a  broader  departmental  context  was  also  a  concern  amongst  the  participants. 
Specifically,  the  CET  was  concerned  about  the  relationship  between  the  CEP  and  the  CBP 
process  and  how  useful  it  would  be  (e.g.,  “A  lot  of  time  was  used  to  understand  how  the  CBP  was 
feeding  our  job”;  “We  struggled  a  lot  on  finding  how  we  could  use  the  [...]  outputs”). 
Throughout  the  exercise,  the  team  spent  a  significant  effort  trying  to  figure  how  the  CEP  fit 
relative  to  the  department’s  overarching  force  development  process. 

Similarly,  a  set  of  departmentally  developed  scenarios  was  to  be  utilized  within  the  exercise  to 
define  and  provide  context  for  its  capability  gap  (goal).  Notably,  however,  out  of  a  total  of  15 
planned  scenarios,  only  five  (5)  were  available  for  use.  Correspondingly,  the  CET  found  it  more 
difficult  than  anticipated  to  delineate  the  capability  gap. 

Timely  understanding.  A  lot  of  direct  and  indirect  evidence  shows  that  the  CEP  construct  was 
difficult  to  understand.  The  participants  did  gradually  understand  the  process,  but  not  in  a  timely 
fashion  (e.g.,  “[We]  understand  the  deliverables  better  after  the  gate  review  as  opposed  to  when 
we  were  producing  them”).  In  the  initial  focus  groups,  the  CET  mentioned  they  did  not  get  the 
‘big  picture’  or  global  perspective.  More  specifically,  they  did  not  understand  the  link  between 
the  work  performed  and  the  force  development  options  which  were  to  be  passed  on  to  the 
decision  maker  at  the  end  of  the  effort  (see  above,  regarding  ‘context’).  Additionally,  there  had 
been  some  confusion  within  the  training  and  documentation  as  to  whether  the  initial  capability 
gap  would  be  given  to  the  team  or  whether  they  would  have  to  define  it.  Consequently,  the  CET 
spent  a  lot  of  time  addressing  this  issue,  despite  no  instructions  to  do  so  (e.g.,  “In  none  of  the 
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training  or  the  deliverables  was  there  anything  on  how  to  define  the  gap”;  “[we  were  taught  that 
we]  will  determine  the  capability  gap  during  the  Inception  phase  [Stage],  but  it  is  considered  an 
input  to  start  it”). 

Comments  from  the  CET  indicated  that  the  Capability  Engineering  construct  has  merit;  however, 
the  best  way  to  achieve  proper  understanding  of  the  CEP  was  not  intuitively  obvious  a  priori.  For 
instance,  one  CET  member  talked  about  the  objectivity  provided  by  the  CEP,  while  another 
member  believed  that  the  CET  should  have  made  better  use  of  the  spiral  approach,  having  been 
willing  to  “move  on  before  having  the  perfect  answer”  (see  above,  regarding  ‘culture’). 
Regardless,  while  there  was  support  for  employing  the  CEP’s  main  foundations,  the  CET  did  not 
feel  they  followed  it  sufficiently  well  to  achieve  key  process  characteristics,  such  as  objectivity 
and  traceability  (“[The]  CEP  gives  objectivity;  the  way  we  did  it  was  not  objective  and  lost 
traceability  -  [there  is  a]  need  to  keep  objectivity  in  terms  of  [the]  process”). 

Iterative  nature.  Participant  comments  confirmed  that  they  did  not  understand  some  key  process 
characteristics.  For  instance,  some  saw  the  process  as  too  sequential,  indicating  that  the 
continuous  consideration  of  performance,  schedule,  risks  and  cost  through  the  ‘Assess  Force 
Development  Options’  technical  sub-process  was  not  properly  understood  (e.g.,  “Go  from 
operational  option  to  systems  option  to  PRICIE.  In  the  end,  there  may  be  constraints  up  front 
(e.g.,  costs)  which  guide  up  front”).  Iterative  progression  was  a  key  CEP  characteristic,  allowing 
for  flexibility  in  producing  deliverables,  with  process  rigidity  concentrated  at  stage  gates  and  in 
terms  of  deliverable  content.  The  CET,  however,  found  these  characteristics  to  be  counter¬ 
intuitive  and  increased  the  challenge  of  applying  the  CEP  (e.g.,  “Iterative  nature  of  the  process 
and  its  flexible  nature  make  it  difficult  to  manage,  both  in  group  and  in  terms  of  documents  and 
templates”).  Further,  some  CET  members  initially  perceived  the  iterative  process  as  representing 
a  waterfall  approach  (“We  ignored  [the]  training  slide  on  [the]  spiral...”)  while  also  finding  its 
application  ‘culturally  challenging’  (“We  could  not  accept  we  didn’t  have  a  perfect  answer  to 
move  on”). 

4.2. 2. 2  Documentation 

The  main  issues  with  respect  to  documentation  were  content  and  expression,  consistency  and 
coherence,  focus  and  approach,  as  well  as  provisioning. 

Content  and  expression.  The  CET  noted  a  lack  of  certainty  and  a  lack  of  guidance  with  respect 
to  where  they  were  in  the  global  CEP  picture.  Specifically,  they  stated  they  were  not  confident 
about  starting  and  finishing  points  within  the  process,  and  that  there  was  a  need  to  routinely 
reinforce  the  global  CEP  context  -  where  they  were,  what  was  done,  what  was  coming  next,  and 
so  forth. 

The  workflow  and  task  descriptions  (provided  both  in  the  CEP  online  and  separately  as  ad  hoc 
documentation)  were  generally  satisfactory.  However,  the  CET  stated  that  there  was  no  guidance 
on  how  to  adapt/customize  the  workflow.  The  workflow  was  also  only  available  to  the  CET  mid¬ 
way  through  the  exercise,  and  was  co-evolving  with  the  process’  templates  (e.g.,  template  and 
task  descriptions  before  and  after  each  iteration  were  not  finalized).  There  was  also  an  unclear 
understanding  of  some  tasks  (e.g.,  incomplete  information),  along  with  timing  and  ordering  issues 
(e.g.,  some  were  too  late,  some  too  early  and  there  was  a  need  to  revise  sequencing  and  definition 
of  certain  tasks). 
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While  the  task  descriptions  identified  artefact  outputs,  the  CET  members  would  have  preferred 
bidirectional  links  between  artefacts  and  their  generating  tasks  (e.g.,  “Need  to  map  architectural 
products  to  workflow”).  Hence,  both  artefact-driven  and  task-driven  CEP  descriptions  are 
required.  Further,  there  was  mixed  preference  for  documentation  to  have  been  expressed  in  terms 
of  (smaller  scale)  artefacts  rather  than  (larger  scale)  deliverables6. 

During  previous  experiments,  the  CET  had  persistently  asked  for  a  workflow  and  a  ‘cookbook 
approach’  to  explain  what  work  needed  to  be  done,  rather  than  replying  on  deliverable  templates 
and  following  an  overview  of  the  process.  As  a  result,  a  workflow  was  developed  and  given  to 
the  CET  (although  only  midway  through  the  exercise);  nonetheless,  its  availability  and  utilization 
within  Exercise  Gamma  did  facilitate  its  validation  by  the  CET.  Curiously,  however,  in  contrast 
to  previous  teams,  Exercise  Gamma’s  CET  preferred  a  dataflow  approach  rather  than  a  workflow. 
Indeed,  some  participants  even  preferred  to  return  to  a  deliverable-centric  approach  instead  of  that 
based  on  a  workflow  (e.g.,  “[We]  should  focus  on  deliverables  [...]  whereas  this  is  focused  on 
activities  and  artefacts”). 

The  expression  of  the  capability  within  the  exercise  was  identified  as  a  significant  issue.  In 
particular,  the  CET  was  confused  by  the  use  of  different  terms  to  refer  to  the  capability  either  as  a 
gap  or  as  a  goal.  Specifically,  exercise  documentation  used  both  terms  in  an  inconsistent  manner. 
Understanding  the  big  picture  was  also  made  more  difficult  due  to  differing  points  of  view  (and 
interpretations)  of  the  varying  terminology. 

The  concept  of  an  ‘Operational  Option’  was  also  a  source  of  confusion  (and  contending  points  of 
view).  This  matter  was  an  issue  up  to  and  including  the  Elaboration  focus  group  (e.g.,  “We  don’t 
understand  how  to  recognize  something  as  an  operational  option”;  “Don’t  see  the  benefit  of 
operational  options...  don’t  understand  what  they  are,  how  they  help.  [Three  CET  members] 
have  different  points  of  view  on  what  is  an  operational  option”;  “Not  recognize  SoS  options  as  an 
operational  option  in  disguise”;  “We  struggled  for  a  long  time  as  to  what  it  means”). 

Commentary  on  the  need  for  metric  documentation  was  also  noted,  at  both  the  Comprehension 
and  Elaboration  stages.  Due  to  timing  and  resource  issues,  the  documentation  only  covered 
metrics  at  the  Elaboration  Stage. 

Consistency  and  coherence.  In  general,  the  main  challenge  in  terms  of  consistency  and 
coherence  was  the  ongoing  evolution  of  the  documentation  in  parallel  with  exercise  execution. 
Additionally,  as  the  CEP  was  also  evolving  at  the  same  time,  there  were  difficulties  in  aligning 
versions  of  the  documentation  to  the  process  as  it  was  being  executed  at  any  particular  moment. 
Furthermore,  issues  of  process  understanding,  such  as  those  mentioned  above,  also  exacerbated 
the  situation. 

As  the  documentation  was  under  development,  the  availability  of  certain  components  was  not 
always  as  desired.  The  detailed  workflow  for  each  stage,  for  example,  was  available  only  at  the 
beginning  of  that  stage.  Therefore,  despite  receiving  an  overview  of  the  whole  process  and  being 
kept  informed  of  its  development,  this  situation  was  not  appreciated  by  some  participants 
(e.g.,  “We  are  working  blind  because  we  have  documentation  on  Comprehension  Stage  and  not 


6 


A  typical  larger  scale  deliverable  would  be  composed  of  various  smaller  scale  artefacts. 
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on  the  Elaboration  Stage”;  “Not  knowing  what  happens  next  makes  it  hard  to  appreciate 
implications”). 

Inconsistency  or  contradictory  information  across  the  various  documentation  sources  (e.g.,  the 
PLA,  CEP  documents,  CEP  online  and  ad  hoc  material  provided  by  the  Evaluation  Team)  were 
reported  in  the  first  focus  group.  Most  documentation  problems  (i.e.,  inconsistencies,  the  need  for 
improvements  and  updates)  were  primarily  due  to  very  little  lead  time  between  confirmation  and 
actual  start-up  of  the  exercise.  Specifically,  there  was  insufficient  time  to  adequately  evolve  the 
documentation  based  on  the  results  from  Exercise  Beta. 

Focus  and  approach.  In  general,  there  was  a  certain  disconnect  in  expectation  between  the  CET 
and  the  philosophy  of  the  CEP.  In  terms  of  task  description,  the  CET  stated  they  would  have 
preferred  a  more  ‘directed’  approach;  specifically,  they  wanted  to  have  detailed  and  exact  task 
information,  more  akin  to  a  ‘meticulous  cookbook’  than  a  sequence  of  expected  results 
(i.e.,  typical  workflow).  The  CET  also  desired  to  follow  a  critical  path  and  looked  for 
unwarranted  meaning  in  the  sample  workflow  (e.g.,  significance  of  its  vertical  axis).  This  desire 
for  a  more  dogmatic  workflow  relates  back  to  the  mentioned  issue  of  process  starting  and 
finishing  points. 

Furthermore,  the  CET  stated  that  the  documentation  needs  to  focus  on  those  things  that  change; 
for  example,  the  “weaknesses  and  tasks  that  need  improvement”  are  those  which  are  repeated 
across  iterations  (versus  singular  components  which  were  well  documented).  Metric 
documentation  also  needed  to  be  similarly  addressed  (e.g.,  according  to  stages).  There  was  also  a 
portion  of  the  CET  that  expressed  the  preference  to  use  project  management  techniques,  such  as 
Gantt  chart  for  deliverables. 

Provisioning.  Feedback  also  addressed  the  ways  and  means  to  provide  documentation  to  the 
CET.  Positive  comments  were  received  on  the  concept  of  an  electronic  version  of  the  CEP,  with 
the  web-based  CEP  documentation  (i.e.,  CEP  online)  being  regarded  as  quite  helpful.  However, 
despite  its  popularity,  two  major  issues  were  noted:  (1)  inconsistency  between  the  online  and 
manuscript-based  documentation;  and  (2)  initial  difficulties  accessing  the  online  version  (due  to 
network  restrictions  and  incompatibilities). 

The  CET  also  indicated  the  need  to  provide  a  better,  less  obtrusive  and  more  transparent  ‘process 
update’  mechanism.  For  example,  a  means  to  ‘update  and  patch’  the  documentation  during 
process  execution  needs  to  be  found.  Other  potential  improvements  or  solutions  include  the  use 
of  databases  (for  increased  accessibility),  stronger  linkage  with  the  tools,  and  reformulation  of  the 
documentation  to  be  “less  like  a  report”  (e.g.,  presented  as  fill-in  templates/paragraphs). 

4.2.2. 3  Process  Constructs 

Three  main  issues  dealing  with  process  constructs  were  highlighted:  iterations  vs.  stages,  serial 
vs.  iterative,  and  breadth  vs.  depth. 

Iterations  vs.  stages.  The  CET  found  the  difference  between  iterations  and  stages  confusing  and 
wanted  to  know  how  to  address  this  issue.  Specifically,  they  requested  clarity  on  how  to 
understand,  manage  and  execute  stages  in  an  iterative  manner,  including  the  impact  this  had  on 
document  creation  and  use  of  templates. 
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Serial  vs.  iterative.  The  CET  stated  they  had  perceived  the  CEP  as  a  ‘waterfall-like’  process, 
and  that  its  “serial  nature  was  a  problem”.  At  the  same  time,  however,  they  stated  they  had 
“ignored  [the]  training  slide  on  [the]  spiral”  because  “they  couldn’t  deal  with  ‘non-perfect’ 
answers”  relative  to  organizational  culture  (see  Section  4.2.2. 1). 

Breadth  vs.  depth.  The  CET  stated  that  in  terms  of  domain  understanding,  their  default 
approach  was  to  go  deeper  before  going  broader  (“. .  .tendency  is  to  dive  too  deep  for  narrow  foci; 
they  want  to  go  deep  before  going  breadth”;  see  Section  4.2.2. 1).  Such  an  approach,  however, 
contradicts  the  iterative/incremental  nature  of  the  CEP. 

4.2.2.4  Deliverables 

The  following  issues  were  the  main  themes  observed  in  terms  of  deliverables  within  Exercise 
Gamma:  relevance,  explicit  responsibility,  ease  of  approach,  domain  content  and  level  of 
satisfaction. 

Relevance.  The  CET  generally  appreciated  the  specified  CEP  deliverables.  Notably,  the  CET 
acknowledged  the  importance  of  the  strategic  and  management  deliverables  that  they  had  initially 
not  deemed  as  important  at  the  beginning  of  the  effort  (e.g.,  “[Strategic  Factor  Analysis']  was  the 
most  important  thing  that  we  did”;  “The  Engineering  Management  Plan  (EMP)  was  more 
important  than  the  CET  had  realized”).  The  deliverables  were  also  appreciated  as  a  means  to 
ensure  traceability  (e.g.,  “Traceability  and  top-downness  are  the  key  things  we  want  to  do  here 
from  the  point  of  view  of  transformation  concepts”). 

Explicit  responsibility.  There  needs  to  be  a  very  explicit  and  well-documented  connection 
between  a  CET  member’s  role  and  responsibilities  and  the  DoDAF  products  on  which  they  will 
work.  While  indicated,  the  connections  were  not  sufficiently  evident  to  the  participants. 
Furthermore,  the  additional,  ad  hoc  documentation  created  by  the  Evaluation  Team’s  process 
advisor  to  address  this  concern  was  regarded  as  insufficient  (e.g.,  incomplete  and  with  too  many 
inconsistencies  and  errors). 

Ease  of  approach.  Due  to  time  pressures  as  the  exercise  neared  completion,  the  CET  shifted 
from  a  task-orientation  to  a  deliverable-oriented  approach,  with  the  importance  of  the  deliverables 
being  made  more  concrete  (e.g.,  “Less  regimented...  I  don’t  want  to  know  about  them 
[i.e.,  tasks].  We  reverted  to  [a]  document  focus”).  While  such  a  change  was  done  in  anticipation 
of  being  easier  (see  Section  4.2.2. 1  regarding  unfamiliar  work  practices),  the  CET  actually  found 
it  difficult  to  do  so. 

Domain  content.  In  terms  of  deliverable  content,  the  CET  was  unable  to  sufficiently  evaluate  the 
utility  of  architectures  within  the  decision  process.  While  the  CET  did  successfully  illustrate  the 
potential  of  architectures  within  the  force  development  process,  they  were  not  able  to  make  solid 
conclusions  about  their  applicability  or  lack  thereof.  However,  the  products  delivered  by  the  end 
of  the  process  showed  that  the  CET  had  produced  all  expected  deliverables,  albeit  sometimes 
more  according  to  the  CET’s  ‘flavour’  than  that  intended  by  the  Evaluation  Team. 


7  The  deliverable  referred  to  as  the  ‘Strategic  Factor  Analysis’  was  technically  known  as  the  Strategic 
Context  Analysis  (SCA). 
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Level  of  satisfaction.  In  terms  of  deliverable  content,  the  CET  seemed  generally  pleased  with 
the  products  they  produced  since  they  provided  an  ‘end-to-end  view’  of  the  problem  and  its 
potential  solutions  (“linking  one  end  to  the  other”).  Some  members  of  the  CET,  however,  would 
have  liked  some  of  deliverables  to  have  had  more  substance  to  them  (i.e.,  quantitative  results,  aka 
“meat”). 

4.2.2.5  Knowledge 

Issues  in  terms  of  knowledge  within  Exercise  Gamma  were  primarily  three-fold:  (1)  missing 
domain  knowledge;  (2)  the  sourcing  and  collection  of  information  from  stakeholders;  and 
(3)  understanding  and  interpretation  of  the  ‘Operational  Option’  concept. 

An  important  issue  for  the  CET  was  sufficient  knowledge  of  effort’s  subject  matter  itself.  The 
team  felt  it  lacked  sufficient  domain  knowledge  (i.e.,  the  definition  of  C4+I);  in  fact,  the  CET  still 
had  concerns  regarding  this  issue  by  the  time  the  Comprehension  Stage  focus  group  was  held 
(e.g.,  “What  is  C4+I,  so  we  can  scope  properly?”).  The  team  spent  a  lot  of  time  trying  to  scope 
the  domain  (e.g.,  “The  functional  decomposition  was  in  effort  to  figure  out  what  C4+I  is;  this 
may  not  be  the  case  in  the  future”). 

The  sourcing  and  collection  of  information  were  also  issues  (e.g.,  “Start  collecting  and  then 
decide  what  to  trim.  I  think  that  the  way  we  collected  information  did  not  work.  We  cannot 
afford  to  wait  until  the  end”).  Moreover,  the  validity  of  information  was  also  an  important  issue, 
such  that  most  of  the  available  information  was  denoted  as  ‘draft,’  thus  making  it  difficult  for  the 
CET  to  feel  certain  about  their  efforts  (e.g.,  “Found  it  problematic  with  everything  being 
DRAFT...  when  can  you  reference/trust  it?”).  The  sourcing  issue  resulted  from  difficulties  in 
being  able  to  sufficiently  liaise  with  the  stakeholders  and  the  appropriate  SMEs  (which  were 
missing  from  the  CET). 

There  were  also  issues  in  terms  of  understanding  the  ‘Operational  Option’  concept.  In  fact,  there 
was  significant  confusion  about  such  a  notion  as  indicated  by  comments  received  up  to  and 
including  the  Elaboration  focus  group  (e.g.,  “We  don’t  understand  how  to  recognize  something  as 
an  operational  option”;  “Don’t  see  the  benefit  of  operational  options...  don’t  understand  what 
they  are,  how  they  help.  [Three  CET  members]  have  different  points  of  view  on  what  is  an 
Operational  Option”;  “...not  recognize  SoS  options  as  an  operational  option  in  disguise”;  “We 
struggled  for  a  long  time  as  to  what  it  means”).  Indeed,  it  was  only  at  the  end  of  the 
Comprehension  Stage  that  the  concept  finally  made  sense  to  the  team.  Further,  there  were  also 
different  points  of  view  as  to  what  an  Operational  Option  actually  was.  While  defined  within  the 
CEP  documentation,  various  individuals  wanted  to  apply  the  process  using  their  own  definition 
(i.e.,  interpretation)  of  the  concept. 

4.2.2. 6  Timeline 

The  issue  of  scheduling  (i.e.,  timeline  and  pacing)  was  also  significant  in  terms  of  the  process 
axis.  Specifically,  the  CET  felt  that  they  were  “...  late  and  lost  in  the  fog”  such  that  one  person 
stated:  “We  saw  the  light  just  in  April  to  understand  what  [it]  was  that  we  needed  to  know.  We 
missed  one  or  two  months”.  As  a  result,  there  was  not  enough  time  to  complete  the  work  within 


DRDC  Ottawa  TR  2011-044 


23 


the  Comprehension  Stage  and  the  CET  could  not  fully  appreciate  the  CEP  by  needing  to  do  it  in 
such  a  condensed  manner  (i.e.,  a  sense  of  urgency  to  complete  this  stage). 
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Figure  6:  Summary  Map  -  Process 
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4.2.3  Materiel 


Challenges  within  the  Materiel  axis  can  be  categorized  in  terms  of  the  following  themes: 
functionality,  logistics,  information  management,  personnel  and  infrastructure.  Aspects  of 
demographics,  performance,  trust,  security  and  timeline  are  also  touched  upon  within  these 
themes. 

4.2. 3.1  Functionality 

The  Exercise  Gamma  CET  utilized  the  ‘standard’  CapDEM  toolset8,  along  with  an  option 
analysis  tool  that  was  introduced  by  one  of  the  CET  members.  Users  benefited  from  the  familiar 
interface  and  intuitive  setup  of  both  the  CEE  and  the  ACCESS  Labs.  In  conjunction  to  a  number 
of  common  tools  and  environments,  a  set  of  specialized  engineering  tools  were  also  part  of  the 
CEE;  these  tools,  however,  were  really  only  used  by  select  members  of  the  CET. 

The  extent  to  which  the  CET  exploited  CEE  functionality  was  influenced  by  the  level  of 
confidence,  comfort  and  training  with  specific  tools,  as  well  as  demographic  considerations 
(ranging  from  age  to  career  track).  Examples  of  a  demographic  effect  include  the  use  of  hardcopy 
projector  tool  (not  used  before  by  any  CET)  as  well  as  the  desire  for  more  desktop  tools  to 
complement  and  integrate  with  physically  shared  facilities  like  the  ACCESS  Labs  (that  is,  the 
creation  of  a  ‘virtual  facility’).  Scheduling  issues  (i.e.,  timeline  and  pacing)  were  also  significant, 
as  the  CET  was  generally  not  able  to  fully  appreciate  or  explore  the  potential  of  the  tools  as  its 
limited  time  and  personnel  resources  were  focused  on  domain  and  process  (CEP)  issues.  As  a 
result,  some  tools  were  used  in  a  limited  and/or  sometimes  incorrect  manner. 

Another  key  issue  influencing  CEE  functionality  was  performance,  such  that  it  significantly 
impacted  the  use  of  (and  trust  in)  many  of  the  CET-wide  general-purpose  tools.  The  Livelink9- 
based  document  repository  and  collaborative  portal/workspace  was  one  such  example:  “Livelink 
did  not  have  performance  needed  -  always  several  seconds  per  click;  cannot  keep  up  with 
people’s  thoughts;  either  the  tool  is  bad,  installed  wrongly,  configured  wrongly  or  [the] 
infrastructure  is  problematic”.  Conversely,  due  to  the  restricted  use  of  some  tools  to  specialized 
users,  the  trust  in  some  technologies  (i.e.,  the  confidence  in  some  tools  and  their  capability)  was 
primarily  achieved  via  reliance  on  specific  users  and  their  experiences.  Consequently,  significant 
functionality  went  largely  unappreciated,  most  often  due  to  performance  problems  resulting  from 
infrastructure  issues  (see  below). 

Functionality  was  also  impacted  by  the  need  to  conduct  (portions  of)  Exercise  Gamma  at  a 
SECRET  classification  level  with  a  CEO  (Canadian  Eyes  Only)  caveat.  Notably,  Exercise 


8  Typical  functionality  provided  by  the  Materiel  axis  throughout  CapDEM  and  across  the  various 
evaluation  exercises  has  included:  requirements  analysis;  functional  analysis;  network  analysis;  human 
systems  integration;  model  repository,  data  integration  and  management;  document  repository  and 
management;  audio  teleconferencing;  video  teleconferencing  (VTC);  collaborative  portal;  shared 
collaborative  workspaces;  e-mail;  and  use  of  various  network) s),  either  as  tool  host  or  communications 
fabric. 

9  Officially,  Livelink  constituted  a  ‘content  management  system’.  Subsequent  to  Exercise  Gamma,  the 
name  ‘Livelink’  was  retired  and  the  product  was  re -branded  as  part  of  the  ‘OpenText  ECM  Suite’  (where 
ECM  stands  for  ‘Enterprise  Content  Management’). 
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Gamma  was  the  first  significant  user  of  the  classified  CEE/ACCESS  Lab,  with  these  facilities  still 
being  under  development  (i.e.,  not  at  ‘production  level’).  Accordingly,  the  CET  felt  that  the 
environment  was  “not  ready  to  deal  with  classified  stuff’.  Further,  the  CEO  caveat  was  seen  to 
stifle  the  ability  to  be  collaborative  due  to  a  lack  of  facilities  (i.e.,  work  places,  available 
networks,  etc.)  which  would  enable  the  desired  levels  of  access,  processing  and  information 
exchange  (for  example,  the  lack  of  a  ‘virtual  facility’).  Such  limitations,  however,  went  beyond 
the  scope  of  CEE  but  related  to  general  policy  (such  as  the  handling  and  consolidation  of  data  and 
information).  The  impact  on  information  exchange  necessitated  the  use  of  varied  means  to 
communicate  and  transfer  information  in  response  to  various  technical,  geographical  and 
security-related  issues  (ranging  from  printed  copy  and  swappable  disk  drives  to  CDs  and  USB 
keys).  In  fact,  this  CET  used  a  hardcopy  projector  as  a  way  to  deal  with  the  inability  to  obtain 
electronic  copies  of  certain  information  (due  to  security  issues).  The  experience  of  transitioning 
to  a  classified  environment  also  impacted  technology  use,  as  the  CET  was  very  cautious  in  terms 
of  respecting  security  considerations  (rules,  regulations  and  practices). 

Infrastructure  issues  (in  particular,  network  infrastructure)  routinely  impacted  the  CET  and  their 
ability  to  use  the  CEE.  Specifically,  the  various  networks  that  were  used  by  the  CET  to 
communicate  and  collaborate  (including  the  DWAN)  were  outside  the  control  of  those  involved 
in  the  exercise.  As  a  result,  any  changes  made  to  such  networks  (by  the  respective  departmental 
authorities)  affected  the  reliability  and  performance  of  CEE,  and  transitively,  its  usage  and 
acceptance  (i.e.,  trust  and  confidence)  by  the  CET  -  both  in  terms  of  specific  tools  as  well  as  the 
broader  environment.  Indeed,  such  changes  were  often  not  announced  and  created  unexpected 
performance  problems  and  application  failure  (in  sometimes  unpredictable  or  misleading  ways). 
Further,  due  to  the  multiple  authorities  involved,  there  was  no  straightforward  way  for  any  of  the 
exercise  participants  (be  in  the  CET  or  the  Evaluation  Team)  to  address  such  issues  (see  the 
upcoming  ‘infrastructure’  section  for  further  discussion). 

The  CET  also  noted  some  concerns  in  terms  of  missing  functionality.  Specifically,  the  CET 
requested:  better  use  and  support  for  configuration  management  (CM),  either  as  a  tool  (as  was 
suggested)  or  in  the  application  of  CM  principles;  the  availability  of  an  automatic  report  generator 
from  process  artefacts  as  part  of  moving  away  from  a  ‘document  orientation’  (i.e.,  requiring  less 
manual  document  creation);  the  addition  of  risk  and  costing  tools,  as  well  as  a  tool  to  fit  the 
‘PRICIE’  structure;  and  the  use  of  an  enterprise  architecture  tool  (at  levels  above  the  CET)  in 
order  to  better  acquire  and  organize  relevant  information  (e.g.,  system  attributes). 

4.2.3.2  Logistics 

Availability  and  access  to  facilities  and  technical  support  proved  a  constant  issue  for  the  CET 
over  the  duration  of  the  exercise.  Issues  of  team  mobility  and  a  desire  for  co-location  versus 
distributed  meetings  were  also  encountered.  Commensurately,  the  CET  also  expressed  the  desire 
to  use  and  be  integrated  into  the  CEE  from  individual  members’  offices,  requiring  secure  remote 
access  to  dedicated  CEE  facilities  (i.e.,  the  ‘virtual  facility’  mentioned  previously).  Similarly, 
there  was  ongoing  demand  for  IT  support  in  terms  of  troubleshooting  the  CEE  as  well  as 
providing  real-time,  on-demand  coaching  and/or  operation  of  the  tools  on  behalf  of  CET 
members.  The  desire  for  more  timely  and  situational-based  training  was  also  noted. 

In  general,  the  varied  logistical  issues  routinely  distracted  the  CET  from  their  focus  and  detracted 
from  their  application  of  the  CEE. 
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4.2.3. 3  Information  Management 


Two  broad  information  management  issues  dominated  the  utilization  of  the  CEE: 
(1)  organization  of  and  access  to  information  for  the  team  to  use;  and  (2)  classification  of 
information  along  with  the  facilities  and  means  to  deal  with  such  material  (e.g.,  transport, 
communication,  storage,  separation,  procedures,  etc.). 

The  knowledge  portal  and  document  repository  (realized  via  Livelink)  was  primarily  used  for 
networked  file  storage  (i.e.,  as  a  shared  drive  replacement)  rather  than  for  collaboration  and  true 
content  (information)  management.  Conversely,  the  CET  requested  support  for  constructs  (such 
as  “versions  of  relationships  between  things”)  that  were  not  directly  supported  by  Livelink  and 
which  would  have  required  significant  effort  to  develop  and  employ  as  part  of  an  appropriately 
structured  information  management  model.  In  the  same  vein,  the  previously  mentioned  tool 
support  for  the  ‘PRICIE’  structure  along  with  the  use  of  an  enterprise  architecture  tool  (at  levels 
above  the  CET)  to  better  enable  information  collection  were  considered  important.  Further, 
various  infrastructural  issues  also  impacted  the  utility  and  application  of  information  management 
practices;  this  issue  is  further  explored  below. 

The  issues  of  information  classification  and  the  means  by  which  to  transport,  communicate  and 
process  said  information  were  overriding  concerns  of  the  CET.  Specifically,  the  CET  wanted  an 
expeditious  means  to  migrate  from  unclassified  to  classified  systems  (i.e.,  between  unclassified 
and  classified  CEE/ACCESS  variants).  However,  as  the  first  significant  user  of  classified 
facilities  (which  were  still  under  development  at  the  time),  the  CET  had  to  address  these  issues  as 
best  possible  without  any  previous  direct  experience  to  call  upon. 

4.2.3.4  Personnel 

As  part  of  evaluating  how  well  a  particular  software  or  hardware  system  met  the  needs  of  a 
particular  usage  context,  both  the  influence  and  experience  of  the  end  user  must  be  addressed. 
Hence,  personnel  considerations  such  as  behaviour  and  attitude  should  be  considered. 

Within  Exercise  Gamma,  the  CET  did  not  utilize  the  functionality  of  certain  tools  in  the  expected 
or  intended  manner.  Additionally,  various  individuals  were  not  interested  in  the  broader  Materiel 
axis,  such  that  they  did  not  want  to  address  CEE  elements  that  were  not  directly  relevant  to  their 
specific  role  and/or  immediate  need.  Some  members  had  difficulty  in  recognizing  the  potential  or 
relevance  of  a  given  tool,  either  with  respect  to  other  tools  or  with  respect  to  how  they  could 
relate  to  the  process  (e.g.,  the  architecture  modelling/analysis  tools).  This  mismatch  and  lack  of 
appreciation  for  functionality  appeared  to  correspond  (to  at  least  some  degree)  to  demographic, 
role/responsibility  and  schedule  influences. 

Similarly,  there  was  a  lack  of  enthusiasm  and  (appropriate)  effort  by  the  CET  to  organize  (or 
learn  how  to  organize)  its  information  in  a  way  that  would  take  advantage  of  the  provided  content 
management  system  (i.e.,  document  repository).  Specifically,  the  CET  sought  ongoing,  real-time 
coaching  for  organizing  documents  along  with  a  dedicated  CET  position  to  address  (i.e.,  perform) 
this  function  in  addition  to  assisting  with  other  tool  and  technology  issues. 

The  issues  of  interest  and  enthusiasm  by  CET  members  also  impacted  the  team’s  level  of  trust  in 
the  various  tools  and  technologies.  Specifically,  as  certain  technologies  were  primarily  used  only 
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by  designated  individuals,  the  perception  of  those  technologies  was  affected  transitively  by  the 
trust  and  confidence  in  their  specific  users. 

4.2.3.5  Infrastructure 

A  key  issue  impacting  the  utilization  of  the  CEE  and  ACCESS  Labs  within  Exercise  Gamma  was 
use  of  multiple  infrastructures  (i.e.,  networks,  facilities,  labs,  etc.).  The  issues  of  availability, 
performance,  predictability  and  reliability  were  of  specific  concern,  and  were  directly  impacted 
by  a  lack  of  control  over  the  multiple  infrastructures  (including  changes  thereto),  insufficient 
awareness  of  the  infrastructures’  varying  technical  requirements,  policies  and  limitations 
(including  changes  thereto),  as  well  as  a  lack  of  personnel  and  expertise  to  address  these  and  other 
technical  issues  resulting  from  the  novelty  of  the  broad  and  uniquely  interconnected  environment. 

The  CET  utilized  a  mix  of  DRDC  and  DND  resources,  both  classified  and  unclassified,  to  share 
information  (as  appropriate  given  security  considerations)  and  collaborate  across  multiple 
geographic  locations  (sometimes  in  an  ad  hoc  manner  ranging  from  offices  and  labs  to  conference 
rooms  and  non-government  facilities).  The  need  to  use  multiple  networks  and  provide  multiple 
entry  points  to  the  CEE  for  purposes  of  location-independent  access  (including  external  Internet 
access)  proved  very  problematic.  Some  difficulties  were  due  to  technical  issues  with  certain 
software  components,  combined  with  a  ‘trust’  issue  between  the  CEE  host  network  and  the  CET’s 
primary  point  of  presence  network10.  In  particular,  the  primary  point  of  presence  network 
implemented  ad  hoc,  unannounced  policy  changes  that  prevented  a  number  of  tools  from 
functioning  correctly  and  creating  significant  performance  problems  when  accessed  across 
networks  (e.g.,  the  Livelink  document  repository /collaborative  portal). 

The  user  experience  was  further  confounded  by  the  mixed  use  of  unclassified  and  classified 
facilities  and  networks,  their  corresponding  policies  and  procedures,  as  well  as  a  lack  of 
familiarity  and  comfort  in  terms  of  their  use.  Due  to  classification,  some  information  could  only 
be  resident  and  processed  on  the  classified  CEE,  while  other  information  could  be  utilized  on 
unclassified  systems.  Therefore,  as  part  of  facilitating  a  flexible  and  user-centric  work 
environment,  the  CET  used  both  categories  of  systems  in  combination.  However,  it  was 
sometimes  difficult  to  be  aware  of  what  information  was  available  on  what  system,  to  maintain 
consistency  in  the  case  of  duplication  between  systems  and  other  information  management  issues 
(as  denoted  above).  Further,  the  creation  of  parallel  classified  and  unclassified  environments 
sometimes  made  it  difficult  for  participants  to  keep  the  rules,  regulations,  policies  and  procedures 
aligned  to  the  correct  environment  (i.e.,  trying  to  remember  what  can/cannot  be  done  in  each 
environment,  along  with  any  differences  in  performing  them).  The  result  was  a  less  than 
confident  user  experience. 


10  For  clarity,  ‘point  of  presence  network’  refers  to  the  main  (unclassified)  network  used  (by  the  CET)  to 
connect  to  the  CEE  for  purposes  of  conducting  the  exercise.  The  ‘CEE  host  network’  refers  to  the  main 
(unclassified)  network  on  which  the  CEE  servers  were  located.  Both  of  these  networks  were  separate  from 
the  classified  CEE  network. 
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5  Discussion 


This  section  now  offers  an  analysis  of  the  various  issues  based  on  the  results  presented  above. 
Recommendations  are  also  put  forward  for  further  consideration  of  best  practices  and  lessons 
learned  to  assist  the  future  institutionalization  of  the  CapDEM  approach.  The  reader  is  also 
referred  to  [9]  and  [10]  for  additional  commentary. 

5.1  CapDEM  Axes 

The  following  section  discusses  the  results  obtained  from  Exercise  Gamma  with  respect  to  each 
axis  (see  Section  4.2). 

5.1.1  People 

Based  on  the  results  presented  in  Section  4.2.1,  the  following  discussion  provides  additional 
commentary  and  analysis  on  issues  pertaining  to  the  People  axis.  Axis-relevant  artificialities  are 
presented  followed  by  considerations  along  the  following  themes: 

•  Team  work,  collaboration  and  communication 

•  Team  charter 

•  Leadership 

•  Roles  and  responsibilities 

•  Contractors 

In  terms  of  CET  experience,  various  artificialities  relating  to  the  effort  being  an  exercise  (rather 
than  an  actual  operation)  should  be  noted.  These  include: 

•  Confluence  of  initiatives.  In  order  to  realize  sponsorship  for  Exercise  Gamma,  a  number  of 
stakeholder  interests  were  coalesced.  Subsequently,  a  variety  of  stakeholder  and  participant 
initiatives  tertiary  to  the  exercise  impacted  the  perspective  and  conduct  of  the  team.  In 
particular,  overlapping  directives,  competing  objectives  and  the  dual  responsibilities  of 
participants  (such  as  to  the  exercise  and  their  home  organization),  resulted  in  Exercise 
Gamma  suffering  from  conflicting  priorities.  Consequently,  the  CET  often  had  to  choose 
between  them  and  in  this  sense,  exploitation  ‘got  in  the  way’  of  evaluation  (“...more 
emotionally  involved  in  the  PLA  than  in  the  CET  initiative”).  In  the  vernacular,  there  were 
basically  ‘too  many  cooks  in  the  kitchen’. 

•  Novelty.  The  uniqueness  of  the  exercise  effort  combined  with  the  novelty  of  the  approach  to 
CET  recruitment  and  newness  of  the  process  made  it  difficult  to  determine  the  driving 
challenges  to  team  organization.  That  is,  it  was  difficult  to  be  certain  if  personnel  issues 
(e.g.,  dedication  and  availability)  were  ‘just  the  way  things  are’  (and  will  always  be),  or  was 
it  specific  to  this  group  of  participants  -  be  it  because  of  the  exercise  designation,  the  first 
significant  external  application  of  CapDEM,  or  the  dual  responsibility  of  participants  (see 
above).  In  general,  inappropriate  recruitment  plus  training  difficulties  were  confounding 
factors  for  other  team  issues. 
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5.1 .1 .1  Team  Work,  Collaboration  and  Communication 


Despite  feeling  they  communicated  well,  the  CET  did  mention  not  feeling  certain  about  how  well 
they  communicated  and  how  well  coordinated  they  were.  In  fact,  they  had  a  difficult  time 
establishing  a  protocol  because  they  hadn’t  figured  out  what  to  do,  and  were  relying  on 
documents  to  ‘lead’  them.  In  the  same  vein,  they  also  wanted  someone  to  use  the  tools  for  them 
(in  particular,  to  perform  tool  and  information  organization).  Consequently,  communication  did 
not  necessarily  lead  to  coordination  in  this  particular  case. 

Indeed,  it  took  a  while  for  the  CET  to  admit  they  had  weaknesses  in  their  communication. 
Meetings  were  considered  as  “boring”  and  while  the  CET  realized  they  lacked  discipline,  they 
never  corrected  themselves.  Subsequent  to  the  introduction  of  the  new  CET  Lead,  who  stated 
“They  didn’t  [even]  know  they  needed  help”,  the  team  changed  the  way  it  worked.  In  particular, 
they  self-organized  in  small  focussed  groups  in  which  it  was  easier  to  work  and  collaborate.  Such 
a  change  in  organization,  however,  underlined  the  need  for  being  more  aware  of  information 
exchange  (between  those  subgroups)  and  created  a  focus  on  communication  issues. 

The  issues  surrounding  communication  and  collaboration  were  also  significant  in  their  effect  on 
the  level  of  trust  within  the  CET.  The  issue  of  trust  is  relevant  to  successful  application  of  the 
CEP  as  iteration  requires  trust  of  other  team  members  due  to  the  lack  of  completeness  (i.e.,  trust 
in  competence  to  provide  suitable  partial  solutions).  It  also  relates  to  not  being  condemned  for 
work  done  along  with  its  perceived  value  at  a  given  point  in  the  process.  Similarly,  there  was  an 
issue  of  members  doing  each  others  work  either  due  to  a  lack  of  communication  and/or  trust  it 
was  done  properly  (i.e.,  confidence).  In  fact,  there  was  a  general  issue  of  being  comfortable  in 
terms  of  commenting  on  others  work  due  to  ill-effective  collaboration;  for  example,  one  member 
did  not  feel  he  was  “really  part  of  the  team”  until  midway  through  the  effort. 

Notably,  the  issue  of  trust  was  mentioned  near  the  end  of  the  exercise,  when  the  CET  was  rushed. 
Therefore,  the  question  is  whether  such  a  concern  was  because  the  CET  was  approaching  a  more 
mature  stage  in  team  development  (i.e.,  ‘norming  vs.  storming’),  or  because  of  a  heightened  sense 
of  urgency.  Indeed,  the  level  of  effort  at  this  time  was  denoted  as  “extremely  involved  and 
intense”,  with  significant  amounts  of  communication  and  working  together.  However,  while  the 
exercise  deadline  did  help  focus  the  team,  it  did  not  necessarily  result  in  the  correct  application  of 
the  process,  nor  did  it  promote  proper  tool  usage.  In  fact,  due  to  tool  usage  patterns,  confidence 
in  the  capability  and  correctness  of  various  tools  was  linked  to  the  trust  in  those  CET  members 
who  were  the  prime  users  of  those  technologies.  Similarly,  confidence  in  the  quality  of  the 
analysis  performed  was  also  related  to  the  level  of  trust  the  CET  had  in  those  members 
performing  that  task.  Consequently,  issues  of  team  work,  collaboration  and  communication  were 
not  only  related  to  technologies  in  their  provision  of  an  enabling  capability,  but  also  in  terms  of 
how  interpersonal  trust  issues  influenced  the  level  of  trust  in  technology.  Trust  in  the  data  used 
was  similarly  influenced,  particularly  given  the  challenges  in  terms  of  SME  access. 

5.1 .1.2  Team  Charter 

The  main  challenge  for  the  team  charter  was  its  lack  of  authority  compared  to  the  other 
documents  governing  the  exercise  (such  as  the  PLA).  While  the  charter’s  contents  were  deemed 
useful,  its  utility  was  ‘weakened’  as  it  was  not  used  as  a  (required)  management  tool.  For 
example,  the  CET  members  did  not  examine  how  it  could  be  used  to  support  team  effort  (e.g.,  no 
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team  discussions  on  rules  of  engagement).  Going  forward,  there  needs  to  be  less  dependency  on 
team  volition  in  terms  of  using  the  charter  and  a  more  defined  link  between  the  team  charter  and 
the  day-to-day  governance  and  functioning  of  the  CET.  That  is,  the  charter  must  be  mandated 
and  used  as  a  practical  and  authoritative  document,  not  as  an  ancillary  ‘good  practices’  reference. 

5.1 .1.3  Leadership 

Leadership  within  the  CET  was  considered  important  by  all  members  in  a  number  of  ways, 
ranging  from  general  aptitude  and  a  willingness  to  listen,  to  being  familiar  with  the  field  (“It’s 
important  that  the  leader  is  knowledgeable  to  the  domain”).  The  individual’s  background  was 
regarded  as  important  as  part  of  being  able  to  make  appropriate  decisions.  It  was  also  stated  that 
the  leadership  needed  to  have  ‘process  expertise’  such  that  a  global  vision  of  the  process  would 
allow  the  team  to  be  effective.  While  these  attributes  are  reasonable,  it  is  important  to  note  that 
such  a  need  was  acute  in  this  exercise  due  to  the  novelty  of  the  effort  and  general  lack  of 
experience  by  the  CET  members.  Future  efforts  that  benefit  from  increased  knowledge  and 
experience  by  all  members  of  the  CET  will  shape  this  requirement  differently. 

Improvement  in  leadership  focus  occurred  in  conjunction  with  a  change  in  leadership  style,  along 
with  a  decrease  in  the  number  of  interruptions  and  unexpected  inputs  from  certain  exercise 
participants  (specifically,  fewer  managerial  inputs  and  certain  contractors).  From  the  start  of  the 
exercise,  the  Evaluation  Team  had  made  a  specific  effort  to  avoid  interrupting  the  CET  so  as  to 
ensure  its  independence  and  avoid  undue  influence.  This  approach  was  in  stark  contrast  to  the 
specific  contractor(s)  who  routinely  interrupted  meetings,  caused  confusion  and  undermined 
leadership.  At  times,  various  contractors  associated  with  the  CET  seemed  more  confident  than 
staff  members,  leading  to  an  awkward  dynamic  on  the  team.  This  dynamic  also  fuelled  criticism 
of  the  Evaluation  Team  for  an  apparent  lack  of  support,  all  the  while  the  Evaluation  Team  was 
trying  to  promote  a  cohesive,  self-sufficient  and  self-lead  team  structure. 

Within  this  exercise,  there  were  too  many  driving  influences  and  subsequent  responsibilities  that 
ended  up  being  addressed  by  the  team  leader  due  to  the  CET  Coordinator  position  not  being 
filled.  As  a  result,  the  CET  Lead  had  to  fill  multiple  roles  which  resulted  in  performing  liaison, 
and  administrative  duties  in  addition  to  producing  the  operational  architecture".  Consequently, 
the  work  load  on  the  individual  in  this  position  was  not  at  the  level  or  as  focused  as  had  been 
intended.  Therefore,  while  the  experience  of  the  CET  Lead  for  Exercise  Gamma  can  be  used  as  a 
reference,  it  cannot  truly  be  considered  commensurate  with  those  likely  in  future  applications.  It 
does,  however,  emphasize  the  importance  of  leadership  as  well  as  suitably  addressing  all 
identified  roles  and  responsibilities. 

5.1 .1 .4  Roles  and  Responsibilities 

As  mentioned  previously,  the  confluence  of  initiatives  surrounding  Exercise  Gamma  (ranging 
from  PLA,  the  exercise  itself,  to  the  Capability  Management  Working  Group)  resulted  in 
confusion  due  to  many  intersecting  initiatives  with  many  (sometimes  overlapping)  leaders  giving 
different  directions.  As  a  result,  people  sometimes  had  ‘divided  loyalties’  (conflicting  priorities) 

1 1  This  list  refers  to  the  final  CET  Lead  who  started  as  the  Operational  Architecture  Lead  and  then  took  on 
some  of  the  CET  Coordinator  functions  (as  that  position  was  not  filled).  Later,  when  the  original  CET 
Lead  departed,  this  person  assumed  that  role. 
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between  the  interests  of  the  various  initiatives  and  their  home  organization.  Consequently,  the 
question  is  whether  such  an  issue  was  more  prominent  because  of  the  effort’s  ‘exercise’ 
designation,  or  whether  is  it  a  challenge  endemic  to  the  use  of  matrix  staffing  in  which  individuals 
are  split  between  multiple  efforts  (e.g.,  ‘dual-hatted’).  For  example,  within  Exercise  Gamma, 
some  participants  asked  why  the  CET  should  care  about  the  PLA,  while  yet  another  stated  he  was 
“. .  .more  emotionally  involved  in  the  PLA  than  in  the  CET  initiative”. 

Cultural  and  personality  influences  relative  to  specific  roles  and  amongst  multiple  roles  were  also 
an  issue.  For  example,  amongst  individuals  and  the  various  organizations  represented  on  the 
CET,  there  was  a  mismatch  in  terms  of  work  styles  and  expectations  on  management.  These 
considerations  also  impacted  how  team  members  accommodated  each  other  and  worked  together. 
For  example,  some  individuals  overtook  other  roles  due  to  the  perception  of  work  not  being  done. 
Such  a  situation  illustrated  a  failure  in  the  team  construct  in  terms  of  trust,  confidence  and 
collaboration.  Subsequently,  the  leadership  was  also  affected  in  terms  of  regulating  such 
behaviour,  as  part  of  ensuring  productivity  and  a  sense  of  team  cohesiveness. 

Another  aspect  of  culture  and  personality  manifested  itself  as  a  result  of  eliminating  the 
‘functional  silos’  which  were  problematic  in  Exercise  Beta  [7].  Unfortunately,  the  solution  -  a 
modified  team  structure  -  led  to  other  issues  in  Exercise  Gamma.  Specifically,  there  was  an 
unanticipated  fixation  on  role  titles  and  their  positions  within  the  org  chart.  In  particular,  certain 
individuals  felt  that  the  title  and  position  implied  importance  within  the  exercise.  Subsequently, 
they  questioned  the  degree  (amount)  of  their  involvement,  and  for  those  involved  in  previous 
efforts,  they  were  concerned  that  they  and/or  the  importance  of  their  work  had  been  ‘demoted’ 
(lessened).  Consequently,  the  use  of  two  sub-leaders  may  not  have  been  the  best  choice  with 
respect  to  org  chart  optics.  Unfortunately,  because  each  CET  will  be  different  (from  focus  to 
member  demographics,  personalities  and  organizational  representation),  there  may  be  an  ongoing 
need  to  suitably  represent  and  explain  the  intent  of  the  team  and  its  structure  in  order  to  avoid 
these  types  of  situations.  Indeed,  this  kind  of  issue  may  be  potentially  addressed  through  a  more 
effective  use  the  team  charter  (see  above). 

Issues  surrounding  specific  positions  within  the  CET,  their  scope  of  responsibility  and  their 
degree  of  importance  were  also  problematic.  This  ranged  from  the  CET  Lead  not  being  clear  as 
to  why  some  of  the  people  in  various  meetings  were  actually  present  (indicating  problems  in 
understanding  and  awareness)  to  being  unclear  if  certain  roles  were  even  necessary.  In  particular, 
the  fact  that  CET  did  not  realize  the  potential  difficulties  that  could  result  from  not  filling  certain 
positions  implies  that  those  roles  were  not  properly  assessed  and  their  functions  rationalized 
(despite  input  from  the  Evaluation  Team).  Notably,  however,  these  perceptions  changed  over 
time  as  the  CET  became  more  familiar  with  the  process  and  the  work  to  be  done.  Consequently, 
it  would  seem  that  the  ‘big  picture’  perspective  of  why  certain  positions  are  part  of  the  CET  needs 
to  be  better  explained  to  and  understood  by  the  CET  and  well  as  any  supporting  organization 
providing  sponsorship,  resources  and  staff.  For  example,  consider  the  following  roles:  CET 
Coordinator  and  Subject  Matter  Experts. 

CET  Coordinator.  This  role  was  added  to  the  CET  structure  based  on  the  results  of  Exercise 
Beta  [7],  and  undeniably,  the  Gamma  CET  did  realize  the  importance  of  the  role  as  the  exercise 
went  along.  In  fact,  the  perception  of  this  role  evolved  considerably,  from  not  being  all  that 
important,  to  then  being  deemed  only  useful  for  administrative  work,  to  then  being  extremely 
important  to  team  function,  and  eventually  being  overwhelmed  even  by  its  administrative 
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component  with  still  more  to  do.  The  lesson  learned  was  not  to  underestimate  the  value  of 
positions  within  execution  of  a  new  process,  particularly  in  the  multidisciplinary  context.  Indeed, 
the  role  definition  was  reasonable  but  the  problem  was  that  the  position  was  not  adequately  filled. 
In  retrospect,  this  situation  highlighted  that  role  nomenclature  (i.e.,  naming)  can  have  unintended 
side  effects  in  terms  of  how  important  it  is  considered.  Specifically,  the  title  this  role  was  given 
resulted  it  being  considered  a  ‘support  position’  (which  is,  unfortunately,  viewed  diminutively); 
consequently,  the  role  did  not  receive  the  attention  it  deserved  and  was  impossible  to  fill  given  the 
current  personnel  challenges  within  the  department.  Indeed,  someone  had  been  assigned  to  this 
position;  however,  he  did  not  show  up  at  initial  CET  meetings,  and  did  not  respond  to  Evaluation 
Team  inquiries  as  to  his  availability.  Subsequently,  and  only  in  response  to  inquiries  through 
uniform  members,  did  he  respond  that  he  was  “too  busy”  to  participate. 

Subject  Matter  Experts.  As  in  Exercise  Alpha  [4]  and  Exercise  Beta  [7],  Exercise  Gamma  had 
to  go  without  (domain)  SME  support12.  While  there  was  much  discussion  as  to  the  importance  of 
SMEs,  there  was  frustratingly  little  support  to  actually  being  able  to  obtain  them.  Indeed,  SME 
availability  remains  a  big  challenge  for  DND.  Moreover,  the  ongoing  question  is  whether  those 
organizations  which  would  benefit  from  the  CapDEM  approach  (such  as  CFD)  will  achieve 
sufficient  import  so  as  to  ensure  access  to  the  required  personnel  and  resources.  To  some  degree, 
such  a  problem  may  be  an  artefact  of  the  effort’s  ‘exercise’  designation,  such  that  the 
‘importance’  of  a  real  operational  context  may  increase  the  willingness  to  participate. 
Furthermore,  the  cultural  considerations  of  the  CET  membership  need  to  be  taken  into  account. 
Specifically,  civilian  scientists  and  engineers  vs.  uniformed  members  typically  have  a  different 
threshold  and  viewpoint  on  engaging  external  personnel  to  address  certain  kinds  of  problems 
(e.g.,  the  military  don’t  like  to  engage  people  prematurely  -  particularly  other  more  senior 
personnel).  Indeed,  the  CET  Lead  stated  he  did  not  feel  he  had  the  authority  to  engage  the  SMEs 
(i.e.,  he  required  an  authoritative  mandate  to  do  so).  However,  this  was  particularly  problematic 
given  his  supplemental  capacity  as  acting  CET  Coordinator,  who  by  definition  was  supposed  to 
be  the  SME  liaison.  Therefore,  aside  from  the  difficulty  experienced  by  the  CET  in  performing 
their  work,  an  important  disadvantage  in  lacking  the  full  CET  (i.e.,  with  SMEs  and  a  CET 
Coordinator)  was  that  it  did  not  allow  for  the  evaluation  of  any  issues  (good  or  bad)  resulting 
from  using  such  an  approach. 

The  issue  of  methodology-based  SMEs  was  also  highlighted  in  the  requests  to  embed  appropriate 
‘technical’  personnel  (i.e.,  CEP  and  CEE  SMEs).  As  opposed  to  domain  SMEs,  this  type  of 
expertise  would  address  the  everyday  operational  concerns  of  CET  in  the  execution  of  their  work. 
Embedding  such  individuals  within  the  team  could  be  an  option  to  mitigate  the  risks  of  the  team 
becoming  ‘lost’  within  the  effort. 

5.1 .1 .5  Contractors 

As  in  many  departmental  initiatives,  contractors  played  a  role  both  within  the  CapDEM  TDP  as 
well  as  Exercise  Gamma.  Within  the  exercise,  those  who  had  focused  tasks  were  generally  more 
appreciated  than  those  who  tended  to  provide  more  ‘free  wheeling’  and  often  officious  input.  In 
the  same  vein,  contractors  seemed  to  be  more  confident  than  the  staff  CET  members  on  the  work 
to  be  done,  leading  to  an  awkward  dynamic  on  the  team.  As  a  result,  some  contractors  appeared 
to  have  had  an  undue  influence  on  the  work  and  the  way  CET  members  did  their  job.  The  CET 


1  The  reference  to  ‘domain  SMEs’  also  includes  the  notion  of  PRICIE  representatives. 
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did  admit  some  responsibility  in  terms  failing  to  corral  contractor  behaviour  in  terms  of  efficiency 
and  suitability  (“The  value  of  contractors  is  providing  them  specific  instructions  with  very  clear 
guidelines,  when  done  and  how  to  do  it  -  we  failed  to  do  that”).  It  would  seem  that  such  an  issue 
related  to  the  CET  not  feeling  in  control,  such  that  they  did  have  the  authority  to  change 
organization  of  the  team  (as  expressed  previously).  Therefore,  it  will  be  important  in  terms  of 
future  efforts  that  the  CET  feels  sufficiently  proficient  and  aware  of  what  is  appropriate  in  terms 
of  capability  engineering  practices  that  they  can  exercise  proper  judgement  in  addressing  team 
issues,  and  specifically  those  involving  contractors.  Doing  so  will  be  an  important  part  of 
managing  and  dealing  with  any  potential  issues  of  bias  or  conflicts  of  interest  amongst 
participants  and  across  concurrent  or  related  initiatives  (see  above  on  ‘exercise  artificialities’  as 
well  as  the  discussion  of  the  Capability  Management  Support  Centre11  concept  in  [10]). 

5.1.2  Process 

Based  on  the  results  presented  in  Section  4.2.2,  the  following  discussion  provides  additional 
commentary  and  analysis  on  issues  pertaining  to  the  Process  axis.  Axis-relevant  artificialities  are 
presented  followed  by  considerations  along  the  following  themes: 

•  Understanding 

•  Knowledge 

•  Documentation 

•  Deliverables 

•  Process  Maturity  and  Assessment 

In  terms  of  CEP  application,  various  artificialities  relating  to  the  effort  being  an  exercise  (rather 
than  an  actual  operation)  should  be  noted.  These  include: 

•  Gate  Review.  Documentation,  organizational  and  rationale  issues  were  the  key  concerns 
affecting  gate  reviews  during  Exercise  Gamma  (in  particular,  the  Inception  gate  review).  In 
terms  of  documentation,  confusion  resulted  from  erroneous  adaptation  of  the  gate  review 
material  (i.e.,  templates)  from  Exercise  Beta.  Once  corrected,  and  the  CET  Lead  and 
Capability  Manager  briefed,  this  issue  was  resolved.  In  terms  of  organizational  and  rationale 
concerns,  the  structure,  organization  and  purpose  of  the  Inception  gate  review  was  not  clear 
a  priori,  such  that  the  CET  did  not  know  who  would  approve  their  work  or  what  would  be 
the  conditions  needed  to  meet  expectations.  Indeed,  such  considerations  would  be  more 
straightforward  in  a  real  application  (e.g.,  with  no  ‘Evaluation  Team’  involved).  However, 
the  issue  highlights  the  general  concern  over  reporting  and  accountability  within  the 
application  of  the  CE  construct.  Indeed,  clarity  on  those  issues  was  required  so  that  the  CET 
could  comfortably  proceed. 

•  Context  and  Readiness.  The  conduct  of  the  effort  as  an  exercise  as  well  as  the  participants 
being  aware  of  its  ‘experimental’  nature  affected  how  the  effort  was  approached.  Further, 
the  unique  governance  of  Exercise  Gamma  resulted  in  its  timeline  being  driven  somewhat 
independently  of  actual  experimental  (e.g.,  Evaluation  Team)  readiness.  These  two  issues 


13  As  of  this  writing,  this  concept  has  evolved  and  the  Capability  Management  Support  Centre  (CMSC)  is 
now  known  the  Centre  for  Capability  Analysis  (CCA)  and  is  being  perused  as  part  of  Project  Accord. 
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intersected  with  cultural  considerations  such  that  larger  departmental  topics  were  paramount 
insofar  that  exploitation  overtook  evaluation  in  the  mind  of  many  participants.  Examples 
included:  dissecting  the  process  relative  to  stakeholder  granularity  instead  of  process 
context  (“...artefacts  [...]  a  lot  are  not  what  VCDS  wants  to  see”);  individual  vs.  team 
orientation  in  terms  of  CBP  correlation,  in  which  individuals  tried  to  link  between  their  job 
and  what  would  be  used  from  CBP  outputs;  and  scenario  importance,  in  which  there  was 
considerable  focus  on  the  state  of  scenarios  as  made  available  from  the  exercise  stakeholder 
(only  5  out  of  15  departmental  scenarios  were  available).  These  topics  will  be  further 
considered  in  the  context  of  process  maturity,  as  discussed  later  in  Section  5. 1.2.5. 

The  lack  of  domain  knowledge  (e.g.,  the  definition  of  C4+I)  and  the  apparently  ineffective 
training  (e.g.,  attempted  to  provide  big  picture  but  “it  didn’t  stick”)  were  additional  examples 
of  insufficient  readiness.  Concern  over  information  validity,  process  documentation  and 
various  CEE  issues  (as  detailed  in  upcoming  sections)  were  also  examples.  The  team  spent 
an  undue  amount  of  time  trying  to  address  some  of  these  concerns,  which  typified  the  unique 
situation  of  the  exercise,  rather  than  what  would  likely  occur  in  actual  operational  situations, 
in  which  participants  would  more  likely  know  the  domain  in  which  they  are  working. 

An  overriding  contextual  issue  was  the  impact  of  ‘meta-influences’  on  the  use  of  the 
process.  Such  influences  are  deemed  ‘meta’  as  they  are  above  and  outside  the  exercise’s 
sphere  of  influence.  For  example,  the  sourcing  of  missing  domain  knowledge  and  collection 
of  information  from  stakeholders  are  critical  to  the  successful  application  of  the  CapDEM 
approach.  However,  the  ability  to  address  these  issues  is  more  dependent  on  the  importance 
and  level  of  commitment  by  CET  members  and  the  department  than  it  is  dependent  on  the 
approach  itself.  That  is,  the  successful  application  of  the  process  depends  on  critical  factors 
outside  of  its  control.  As  Exercise  Gamma  was  a  trial  application,  the  level  of  support  by  its 
CET  and  stakeholders  (i.e.,  the  broader  department)  was  (in  their  opinion)  commensurate  to 
that  context.  Unfortunately,  such  a  difference  only  permits  an  approximation  of  true  usage 
context  and  thus  impacts  its  evaluation  accordingly. 

5.1 .2.1  Understanding 

As  introduced  in  4.2.2. 1,  a  combination  of  direct  and  indirect  evidence  illustrated  that  the  CEP 
construct  was  difficult  to  understand,  particularly  in  a  timely  fashion.  The  CET  stated  they  often 
felt  as  if  they  were  “lost  in  the  fog”  and  struggled  to  understand  the  effort  in  its  entirety  for  nearly 
half  the  experiment.  As  a  result,  the  CET  fell  behind  schedule  and  needed  to  rush  during  the  final 
stages.  Indeed,  they  developed  a  sense  of  urgency  (akin  to  cramming  for  an  exam)  that  resulted 
in  higher  productivity  within  the  last  E/2  months  than  that  achieved  up  to  that  time.  As  the  CET 
finally  saw  the  potential  of  the  process  and  its  iterative  nature  by  the  end  of  the  exercise,  it  is 
believed  that  if  the  same  CET  were  to  perform  a  second  instance  of  the  process,  they  would 
produce  significantly  less  encumbered  results,  and  that  a  large  number  of  the  problems 
encountered  wouldn’t  occur  due  to  increased  understanding  from  experience.  As  it  was, 
however,  CET  did  not  (and  could  not)  fully  appreciate  the  CEP  by  doing  it  in  such  a  condensed 
manner. 

Based  on  the  feedback  received,  it  was  realized  that  the  provided  training  had  failed  to  instil  the 
‘big  picture’  to  the  CET.  That  is,  despite  providing  a  global  overview  at  every  training  session, 
including  where  the  CET  was  relative  to  the  overall  effort,  what  had  worked  to  that  point,  and 
what  was  coming  next,  the  CET  had  a  difficult  time  understanding  how  the  work  they  performed 
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would  ultimately  result  in  force  development  options  which  would  be  relevant  to  decision 
makers.  Commentary  as  to  whether  all  CEP  products  were  “what  the  VCDS  wants  to  see”  was  a 
sign  of  immaturity  in  process  application  (not  the  process  itself)  and  a  failure  to  see  how  various 
levels  of  products  would  culminate  in  the  appropriate  manner  through  the  appropriate  process 
constructs. 

Undeniably,  the  CET  did  not  understand  some  key  process  constructs  and  characteristics.  For 
instance,  some  members  saw  the  process  as  too  sequential,  indicating  that  the  continuous 
consideration  of  aspects  such  as  performance,  schedule,  risks  and  cost  through  the  ‘Assess  Force 
Development  Options’  sub-process  were  not  understood.  In  spite  of  that,  some  members  didn’t 
think  that  cost  constraints  were  considered  early  enough,  while  yet  others  criticized  the  CEP 
Capability  Engineering  Decision  Framework  (CEDF)  for  reducing  the  problem  “to  just  money”. 
Other  examples  included: 

•  Iteration  vs.  stage:  Confusion  over  which  construct  was  which  and  how  they  related  to  each 
other  was  encountered.  Specifically,  the  CET  needed  clarity  on  how  to  execute,  manage  and 
understand  stages  in  an  iterative  manner.  One  member’ s  perception  of  performing  the  entire 
series  of  stages  within  each  iteration  was  an  example  of  such  a  misunderstanding.  Going 
forward,  this  kind  of  relationship  and  its  progression  must  be  more  clearly  communicated. 

•  Workflow:  By  design,  there  was  not  really  a  critical  path  through  the  CEP.  As  such,  the 
CEP  was  not  intended  to  be  a  prescriptive  ‘cookbook’  but  provided  the  CET  the  freedom 
and  the  responsibility  to  progress  their  own  work  using  the  stages  and  iterations  based  on  a 
series  of  expected  outputs.  The  attempt  to  utilize  a  previous  effort’s  workflow  was 
problematic  as  it  was  too  specific  to  the  given  question  and  technical  environment  they  used. 
Going  forward,  workflows  should  be  specified  suitably  orthogonal  to  (i.e.,  independently  of) 
the  specific  problem  and  technical  environment.  This  topic  will  be  discussed  further  in  the 
upcoming  Materiel  section. 

•  Depth  vs.  breadth:  Part  of  the  reason  that  the  CET  found  it  difficult  to  utilize  the  CEP  was 
their  predilection  to  address  topics  more  in  terms  of  depth  than  in  terms  of  breadth 
(“...tendency  is  to  dive  too  deep  for  narrow  foci;  they  want  to  go  deep  before  going 
breadth”).  While  such  an  approach  is  a  cultural  tendency  within  the  organization,  it 
contradicts  the  inherent  iterative  and  incremental  nature  of  the  CEP,  and  will  therefore 
require  concerted  effort  to  adopt  such  a  way  of  working.  Indeed,  it  will  be  a  function  of 
experience  and  maturity  in  the  application  of  the  process  that  will  enable  an  increasingly 
smoother  and  more  natural  application  of  the  approach  across  an  increased  range  of  topic 
areas. 

•  Iterative  progression:  The  iterative  nature  of  the  CEP  can  make  its  management  and 
application  somewhat  challenging.  One  intended  benefit  is  flexibility  in  the  production  of 
deliverables,  such  that  they  can  be  developed  incrementally  as  a  whole  set,  in  relative 
concert  with  each  other.  Equally,  the  early  identification  of  non-viable  options  as  the 
iterations  progress  allows  work  on  infeasible  alternatives  to  be  suspended  in  a  timely  manner 
and  future  effort  to  be  focused  on  any  remaining  options.  Indeed,  both  situations  gain  from 
the  ability  to  get  feedback  at  multiple  points  in  the  process.  Attaining  such  an  end,  however, 
requires  the  discipline  to  continuously  synchronize  the  advancement  of  deliverables  rather 
than  necessarily  taking  any  particular  output  to  completion  in  advance.  The  result  is  a  series 
of  intermediate  products  that  are  a  progression  to  the  final  deliverable  (much  like  the  ‘track’ 
of  a  moving  vessel  towards  its  eventual  destination).  Within  Exercise  Gamma,  however,  the 
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CET  members  had  a  difficult  time  “letting  go”  and  “moving  on”  such  that  further 
information  would  be  fleshed  out  later  (“We  could  not  accept  we  didn’t  have  a  perfect 
answer  to  move  on”).  This  reluctance  was  seemingly  influenced  by  issues  of  work  norms, 
trust  and  culture.  Based  on  the  CET  demographics,  the  work  norms  of  its  members  were 
such  that  they  had  a  high  desire  for  accuracy  and  correctness  (associated  with  technical  and 
military  backgrounds).  When  combined  with  a  lack  of  familiarity  with  the  process,  facets  of 
the  ‘not  created  here  syndrome’  were  manifest  such  that  the  team  decided  to  do  things  their 
own  way  (“...  ignored  [the]  training  slide  on  [the]  spiral...”  as  they  “couldn’t  deal  with 
‘non-perfect’  answers  in  DND  culture”).  Cultural  issues  of  trust  and  accountability  were 
evident  in  that  team  members  did  not  necessarily  feel  comfortable  relying  on  interim 
products  from  other  members  (i.e.,  did  not  trust  others  to  have  materials  ready  as 
appropriate),  especially  given  the  apprehension  of  potentially  being  held  responsible  for 
incorrect  or  prematurely  released  information/analysis.  The  issue  of  trust  would  seemingly 
stem  from  a  lack  of  maturity  in  terms  of  the  process  (i.e.,  being  a  new  construct),  its 
application  (i.e.,  a  lack  of  experience  with  the  construct)  and  as  a  team  (i.e.,  a  lack  of 
familiarity  with  new  members,  specifically  their  personalities,  work  practices  and  ethics). 
Concerns  over  accountability  reflect  a  cultural  predisposition  to  ‘denounce’  those 
responsible  for  work  done  when  difficult  situations  arise.  Such  concerns  being  notable  even 
in  an  exercise  context  were  problematic  for  the  team  both  in  terms  of  input  and  output.  As 
input,  the  CET  was  frustrated  by  only  being  able  to  obtain  ‘draft’  information;  as  output, 
there  was  reticence  that  iterative  versions  may  be  incorrectly  used  and  attributed.  Such 
anxiety  regarding  inappropriate  use  of  premature  results  (i.e.,  early  iterations)  will  need  to  be 
overcome  in  order  to  take  advantage  of  the  intended  benefits  of  the  CEP.  Similarly,  the 
intersection  of  these  issues  between  the  People  and  Process  axes  highlights  the  need  for 
further  research  into  how  to  provide  better  support  for  such  constructs  within  the 
organization  and  its  approach  to  staffing. 

Representation:  The  physical  representation  of  the  process  may  have  added  to  the  difficulty 
in  understanding  the  process,  due  to  the  inherent  challenges  in  representing  an  iterative, 
incremental  effort  in  a  flat,  static  diagram.  Similarly,  the  process  documentation  (such  as  the 
templates)  was  not  necessarily  intuitive  in  such  a  context  (e.g.,  “Iterative  nature  of  the 
process  and  its  flexible  nature  make  it  difficult  to  manage,  both  in  group  and  in  terms  of 
documents  and  templates”).  Consequently,  there  is  a  need  to  avoid  the  process 
representation  from  negatively  influencing  its  application  while  also  finding  a  more  suitable 
means  to  represent  a  dynamic  and  multi-level  process.  This  topic  is  further  discussed  in  the 
upcoming  section  on  documentation. 

Capability  gap:  While  the  targeted  capability  (be  it  a  gap  or  a  goal)  is  a  fundamental  driver 
for  the  CapDEM  approach,  its  determination  was  not  intended  to  be  part  of  the  exercise. 
Nevertheless,  the  CET  spent  a  significant  amount  of  time  addressing  this  issue  along  with 
various  others  not  intended  to  be  part  of  the  exercise.  This  foray  was  primarily  rooted  in  the 
following: 

♦  Terminology  and  interpretation:  Despite  attempts  to  rationalize  any  difference  in 
terminology,  the  lack  of  consistency  within  the  documentation,  training  and  amongst 
participants  in  referring  to  the  capability  as  a  gap  or  as  a  goal  created  confusion  for 
the  CET.  Further,  it  was  noted  that  different  interpretations  are  possible  with  respect 
to  these  terms,  given  the  variability  in  their  expression  (both  inside  and  outside  the 
department). 
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♦  Perspective:  As  various  personnel  involved  in  Exercise  Gamma  were  also  involved 
in  related  efforts  (such  as  the  CMWG),  their  impressions  were  sometimes  coloured 
and  behaviour/outputs  influenced  as  a  result  of  their  dual  roles  and  the  different 
perspectives  between  those  efforts  and  the  CEP  (see  Section  5.1.E4). 

♦  Difficulty:  The  capability  gap  for  Exercise  Gamma  was  not  easy  and  some  CET 
members  put  an  emphasis  on  having  the  complete  gap  understood  before  even 
starting  to  think  about  operational  options.  As  the  capability  gap  was  provided  to 
the  CET  through  the  use  of  a  CFD-provided  scenario,  the  limited  availability  of  such 
scenarios  had  a  significant  impact  (as  the  CFD  CBP  team  had  only  completed  5  out 
of  15  scenarios).  In  an  actual  operational  context,  these  considerations  would  be 
more  suitably  defined  (i.e.,  less  artificial). 

♦  Focus:  While  the  relationship  between  the  CEP  and  the  department’s  CBP  process 
is  clearly  important,  the  CET  was  sometimes  overly  concerned  about  the  relationship 
and  how  useful  it  would  be.  In  particular,  their  focus  on  the  CEP  proper  was 
sometimes  diluted  relative  to  concern  over  the  CEP  interface  with  other 
(departmental)  processes.  It  is  anticipated  that  this  issue  will  tend  to  be  of  less 
concern  in  the  future  given  increased  experience  in  applying  the  process,  resulting  in 
increased  understanding  of  the  CEP  and  the  interface  to  other  processes. 

5.1 .2.2  Knowledge 

The  availability,  sourcing  and  collection  of  domain  knowledge  are  critical  enablers  for  the 
successful  application  of  the  CapDEM  approach.  Consequently,  the  results  identified  in 
Section  4.2. 2. 5  merit  further  consideration  given  the  previously  outlined  influences  and  issues 
affecting  process  understanding. 

•  Operational  Option.  While  some  inconsistency  in  the  understanding  of  an  ‘Operational 
Option’  may  have  been  in  part  due  to  documentation  issues  (see  next  section),  the  primary 
problem  was  that  various  individuals  wanted  to  apply  the  process  using  their  own  definition. 
As  denoted  in  Section  5. 1.2.1,  individuals’  perspective  and  understanding  of  key  process 
elements  were  influenced  by  their  various  ancillary  efforts.  In  the  same  vein,  the  various 
changes  in  team  leadership  created  the  opportunity  for  such  divergence  to  become 
established  within  the  group.  Hence,  a  general  lack  of  experience  (i.e.,  confidence)  by  the 
CET  combined  with  a  lack  of  SMEs  created  an  opportunity  for  such  confusion  to  thrive. 

•  Subject  Matter  Experts.  The  role  of  subject  matter  experts  (SMEs)  as  knowledge  providers 
within  the  CapDEM  approach  can  generally  be  categorized  in  two  ways:  (1)  knowledge  as 
the  basis  for  analysis;  and  (2)  knowledge  as  part  of  facilitating  the  analysis. 

♦  Basis  for  analysis.  This  category  of  SMEs  is  intended  to  provide  access  to  ‘real 
world’  operational  expertise  in  terms  of  information  to  be  used  by  the  CET  within 
application  of  the  CEP  (e.g.,  PRICE  representatives).  This  information  constitutes 
the  domain  knowledge  on  which  the  analysis  is  based.  The  availability  of  that 
knowledge  (and  thus  access  to  the  associated  expertise)  is  essential  to  ensure  that  the 
information  being  processed  is  both  suitable  and  valid.  Notably,  because  so  much  of 
the  information  available  for  use  in  Exercise  Gamma  was  denoted  as  ‘draft’,  the 
CET  felt  apprehensive  about  the  reliability  of  their  results.  Difficulties  in  liaising 


DRDC  Ottawa  TR  2011-044 


39 


with  the  stakeholders  and  the  appropriate  SMEs  further  exacerbated  those  concerns, 
often  leaving  the  CET  feeling  frustrated. 

♦  Facilitation  of  analysis.  Due  to  the  various  challenges  and  issues  in  applying  the 
CEP  and  utilizing  the  CEE,  there  was  a  need  for  SMEs  to  provide  ongoing,  real-time 
support  and  coaching  in  terms  of  the  process  and  tools.  The  manner  in  which  to  best 
provide  such  expertise  was  the  subject  of  debate  throughout  the  exercise  (see 
sections  4.2.1,  4.2. 3.4,  5. 1.1. 4  and  5. 1.3.5).  However,  due  to  logistical  and 
personnel  issues,  a  satisfactory  level  of  support  was  never  reached.  This  issue 
consequently  resulted  in  frustration  amongst  not  only  the  CET,  but  the  Evaluation 
Team  as  well. 

•  Separation  of  knowledge  from  process:  Due  to  the  novelty  inherent  in  so  many  aspects  of 
the  exercise,  it  was  often  difficult  to  clearly  distinguish  the  source  of  particular  problems.  In 
particular,  while  process  and  knowledge  are  inextricably  linked,  they  are  not  the  same.  It  is 
generally  difficult  to  discern  the  difference  during  early  application  of  the  process  due  to  a 
lack  of  experience  of  what  is  process  and  what  is  driving  the  process.  While  such  a 
consideration  is  not  particular  to  the  CEP  (and  is  indeed  a  much  broader  issue),  it  is  key  to 
realizing  that  the  inability  to  address  domain  knowledge  should  not  be  misattributed  as  a 
process  failing.  That  is,  as  mentioned  in  the  introduction  to  Section  5.1.2,  meta-level 
influences  must  be  appropriately  isolated  in  order  to  accurately  identify  any  problematic 
process  issues. 

5.1 .2.3  Documentation 

Documentation  considerations  generally  fit  in  one  of  two  categories:  (1)  content  and  expression; 
and  (2)  focus  and  approach. 

Content  and  expression.  In  terms  of  content  and  expression,  the  issues  of  most  concern  were 
related  to  workflow,  tasks  and  update  logistics. 

•  Workflow.  As  mentioned  previously,  there  was  not  really  a  ‘critical  path’  within  the  CEP, 
as  it  was  not  intended  to  be  a  directed  (i.e.,  prescriptive)  process;  rather,  it  was  intended  to 
outline  the  sequence  of  expected  results,  giving  the  CET  the  freedom  and  the  responsibility 
of  managing  their  own  work  (i.e.,  both  descriptive  and  deliverable-centric).  Conversely,  the 
CETs  from  the  previous  experiments  (in  particular,  Exercise  Beta)  had  repeatedly  asked  for 
a  workflow  and  a  ‘cookbook  approach’  to  direct  their  efforts. 

Consequently,  a  workflow  was  developed;  however,  it  was  completed  only  midway  through 
the  exercise.  While  this  scheduling  did  allow  the  workflow’s  suitability  (correctness)  and 
utility  to  be  validated  by  the  CET  through  application  in  the  latter  half  of  the  exercise,  the 
CET  did  not  appreciate  its  late  availability.  Some  members  considered  the  workflow 
insufficient,  open  to  misinterpretation,  and  somewhat  ambiguous  (i.e.,  they  would  have 
preferred  a  previously  vetted  workflow).  Unfortunately,  the  previously  validated  workflow 
was  tied  too  tightly  to  an  earlier  incarnation  of  the  CEE  (see  ‘workflow’  under 
Section  5. 1.2.1),  reaffirming  the  necessity  of  orthogonality  in  its  specification. 

Contrary  to  earlier  feedback,  some  participants  within  Exercise  Gamma  subsequently 
expressed  a  preference  to  return  to  a  deliverable-centric  approach  instead  of  one  based  on 
workflow.  Such  variation  in  commentary  both  for  and  against  workflows  confirms  that 
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different  CETs  have  had  -  and  will  continue  to  have  -  different  ways  of  working  and 
approaching  the  process.  Indeed,  both  deliverable-centric  and  task-centric  methods  are 
complementary  means  of  understanding  the  process,  and  going  forward,  support  for  both 
will  foster  greater  understanding  and  acceptance  of  the  process  within  the  department.  It 
will,  however,  make  documentation  implicitly  more  difficult  due  to  the  need  to  ensure 
consistency  and  continuity  between  perspectives. 

As  part  of  facilitating  both  approaches,  an  ‘artefact-centric’  methodology  was  proposed.  As 
artefacts  are  generally  self-contained,  fine  granularity  information  fragments 
(e.g.,  paragraph,  figure  or  table)  which  together  compose  a  document,  both  deliverables  and 
tasks  can  utilize  the  same  fragment  (i.e.,  artefact).  The  potential  of  such  an  approach  is  to 
enable  increased  automation  in  terms  of  deliverable  composition.  Doing  so  helps  maintain 
consistent  and  complete  documentation,  in  which  less  effort  is  spent  on  reporting 
administration  and  more  on  artefact  content  (“...make  it  less  like  writing  a  report...”). 
However,  an  artefact  orientation  has  a  significant  effect  on  tool  use  and  information 
structure,  requiring  more  customization  and  tighter  integration  between  all  three  axes  than 
was  possible  during  the  exercise  (see  the  Materiel  section  for  additional  commentary). 

•  Tasks.  The  main  considerations  in  terms  of  tasks  were  their  understanding,  scheduling  and 
description.  The  concurrent  evolution  of  task  definition,  sequencing  and  their 
documentation  constructs  (i.e.,  templates)  made  understanding  and  scheduling  difficult. 
Indeed,  one  source  of  such  difficulties  was  that  while  a  task  could  be  performed  within  the 
bounds  of  multiple  iterations,  there  was  only  one  description  provided  within  the 
documentation.  Further,  the  evolution  of  the  CEP  during  the  course  of  the  exercise  meant 
that  the  CET  had  to  deal  with  multiple  versions  of  the  documentation  constructs 
(e.g.,  templates  and  task  descriptions).  Additionally,  task  descriptions  were  also  complicated 
by  the  CET’s  desire  for  exhaustive  detail  on  each  task,  the  desire  for  focus  on  aspects  which 
changed  between  iterations  along  with  the  early  provision  of  metrics  (starting  at  the 
Comprehension  Stage). 

•  Update  logistics.  Issues  of  how  to  disseminate  modifications  to  the  process  and  its 
documentation  were  of  concern  to  the  CET,  including  being  able  to  more  easily  ‘update  and 
patch’  in  situ  (i.e.,  mid-execution),  so  as  to  support  future  multiple  and  concurrent 
applications.  To  this  end,  the  introduction  of  electronic  documentation  (via  the  ‘online 
CEP’)  was  well  received.  Regarded  as  easier  to  use  than  a  paper-based  approach,  it  is  also 
more  amenable  to  future  modifications,  including  in-place  updates. 

In  general,  however,  it  is  worth  noting  that  some  of  the  complaints  received  during  the  exercise 
seemed  misattributed  and/or  ‘unjustified’,  ostensibly  stemming  from  difficulties  in  understanding 
and  documentation.  Indeed,  future  operational  incarnations  of  the  CapDEM  approach  will  not 
likely  suffer  from  the  artificialities  the  exercise,  or  its  constricted  development  timeline.  For 
example,  some  users  “felt  blind”  because  certain  documentation  components  were  not 
immediately  available.  Rather,  increased  application  within  the  department  and  being  more 
solidly  rooted  in  an  organizational  structure  would  naturally  correspond  with  higher  availability, 
increased  consistency  and  fewer  incongruities  across  documentation  sources. 

Focus  and  approach.  In  terms  of  focus  and  approach,  the  issues  of  most  concern  were  related  to 
orientation  and  philosophy.  It  is  again  noteworthy  to  emphasize  the  impact  that  orientation 
(i.e.,  artefacts  vs.  deliverables  vs.  tasks)  will  have  on  technological  support  (i.e.,  tools),  as  well  as 
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the  composition  of  documentation  and  outputs.  The  desire  for  increased  automation  through  the 
use  of  report  generators  and  artefacts  needs  to  consider  the  balance  between  ease  of  use  and 
productivity  versus  the  freedom  and  flexibility  of  a  more  manual  approach.  Similarly,  the 
philosophical  difference  as  how  to  best  approach  the  process  (i.e.,  descriptive  vs.  prescriptive) 
also  influences  the  level  of  detail  provided  versus  that  desired  by  some  participants.  To  what 
degree  future  versions  of  the  process  will  need  to  balance  these  two  approaches  is  unclear  at  this 
point  and  requires  further  consideration  based  in  part  on  anticipated  demographics  (i.e.,  cultural 
and  experiential  perspectives)  of  future  CETs.  This  philosophical  consideration  also  relates  to  the 
desired  ability  to  customize  the  CEP  (and  its  workflow).  While  such  a  feature  will  be  desirable  as 
the  process  becomes  more  widespread,  premature  use  of  customization  in  lieu  of  proper 
understanding  and  application  will  negatively  influence  the  process’  perceived  reliability. 

5.1 .2.4  Deliverables 

Issues  in  terms  of  deliverables  generally  fit  in  one  of  three  categories:  (1)  relevance;  (2)  process 
outputs;  and  (3)  content  and  satisfaction. 

Relevance.  By  the  end  of  the  exercise,  all  expected  deliverables  had  been  produced  by  the  CET. 
While  the  character  of  some  deliverables  deviated  from  what  was  intended,  the  specified 
deliverables  were  appreciated  by  the  CET,  both  in  terms  of  ensuring  traceability  as  well  as 
providing  for  an  ‘end-to-end  view’  (see  Section  4.2.2. 4).  The  CET  ultimately  understood  the 
potential  of  the  process  and  its  iterative  nature,  despite  having  to  struggle  with  it  for  nearly  half  of 
the  exercise.  Notably,  there  was  general  agreement  (both  amongst  the  CET  and  the  Evaluation 
Team)  that  should  the  same  CET  perform  a  second  application  of  the  process,  their  efforts  would 
be  significantly  less  encumbered. 

Process  outputs.  The  CET  desired  a  very  explicit  connection  between  a  member’ s  responsibility 
and  the  products  they  ‘needed  to  touch’  (i.e.,  produce,  use,  amend,  etc.).  While  such  connections 
were  documented,  the  level  of  explication  provided  was  regarded  as  insufficient  and  there  was  a 
strong  negative  reaction  to  any  errors.  In  general,  the  clarity  of  deliverables  and  their 
documentation  were  an  issue  of  process  maturity,  such  that  alignment  between  team  member  and 
outputs  (deliverables)  would  naturally  evolve  as  they  are  validated  over  time.  Nonetheless,  CET 
feedback  spoke  to  immediate  concerns,  based  in  part  on  their  experience  within  the  exercise.  For 
example,  as  the  CET  started  to  run  out  of  time  to  complete  the  exercise,  they  shifted  from  a  task- 
based  approach  to  a  deliverable-oriented  one,  thinking  that  it  would  be  easier.  However,  in 
reality,  that  approach  was  also  difficult  to  execute  and  they  continued  to  feel  under  pressure.  In 
the  objective  sense,  this  situation  illustrates  that  there  is  no  perfect  representation  of  the  process 
and  that  the  level  of  clarity  needs  to  be  inversely  proportional  to  the  level  of  familiarity  with  the 
work  practice  (i.e.,  the  less  familiar  the  work  practice,  the  clearer  the  explication  needs  to  be  to 
avoid  increased  resistance  to  that  portion  of  the  process). 

Content  and  satisfaction.  As  Exercise  Gamma  was  intended  to  provide  results  as  a  trial 
application  (not  just  as  an  evaluation  exercise),  the  content  of  the  deliverables  was  evaluated  from 
a  broader  perspective,  considering  both  client  and  participants.  As  per  Section  4.2. 2.4,  CET 
members  were  generally  pleased  with  the  products  produced,  but  they  would  have  preferred  more 
substance  (i.e.,  detail).  Notably,  this  preference  varied  within  the  team  and  was  generally  aligned 
with  cultural  and  demographic  groupings.  Going  forward,  the  merit  of  creating  benchmarks  for 
deliverable  completion  and  how  to  provide  guidance  in  support  of  such  benchmarking  remain 
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open  questions.  In  terms  of  client  goals,  the  exercise  did  illustrate  the  potential  of  architectures 
and  capability  engineering  within  the  force  development  process;  however,  the  CET  was  not  able 
to  make  solid  conclusions  about  their  applicability  within  the  decision  process.  Nonetheless, 
satisfaction  was  expressed  with  the  outcome  of  the  trial  application  and  the  learning  that 
resulted  [14].  As  noted,  in  order  to  accurately  demonstrate  the  utility  of  architectures  and 
capability  engineering  within  the  decision  process,  a  broader  comparative  study  would  be 
required,  not  just  a  single  ‘one  off’  application.  Regardless,  the  exercise  was  regarded  as  an 
important  and  necessary  step  in  that  direction. 

5.1 .2.5  Process  Maturity  and  Assessment 

A  number  of  previous  topics  merit  consideration  in  terms  of  process  maturity  and  their  effect  on 
assessment.  Such  overarching  issues  provide  a  lens  for  examining  the  particularities  of  the 
exercise  in  a  more  holistic  manner. 

As  the  last  of  three  evaluation  exercises,  Exercise  Gamma  benefited  from  the  experiences  and 
lessons  learned  from  both  Exercise  Alpha  and  Exercise  Beta.  That  being  said,  the  cultures  of  all 
three  evaluation  exercises  were  substantially  dissimilar,  significantly  due  to  very  different  CET 
composition.  The  cultural  influences  were  in  part  significant  because  they  impacted  the  conduct 
of  the  exercises,  ranging  from  CET  interaction  to  CEP  interpretation  as  well  as  CEE  application. 
In  terms  of  the  CEP,  cultural  influences  affecting  the  interpretation  of  ‘technically  objective’ 
processes  are  specifically  challenging  when  the  process  is  descriptive  rather  than  prescriptive  (see 
Section  5. 1.2. 3)  and  there  is  a  general  lack  of  experience  with  the  process  being  applied.  That  is, 
in  the  case  where  there  is  little  previous  experience  to  draw  upon  in  terms  of  applying  the  process, 
the  interpretation  and  opinion  of  the  practitioners  can  be  particularly  influential.  Further,  because 
of  this  heightened  subjectivity,  team  structure  (e.g.,  demographics)  and  dynamics 
(e.g.,  personalities)  can  have  an  unintended  level  of  influence  on  CET  conduct. 

For  example,  both  a  degree  of  impatience  and  a  degree  of  obstinacy14  were  encountered 
throughout  the  exercise.  In  particular,  the  determination  to  address  specific  issues  and  avoid 
interactions  with  relevant  individuals  embodied  obstinacy:  the  CET  spent  a  lot  of  time  addressing 
issues  not  intended  to  be  part  of  the  effort  (see  ‘capability  gap’  under  Section  5. 1.2.1)  and  did  not 
want  ‘to  bother  people’  by  asking  questions.  Similarly,  members  of  the  CET  had  diverse 
opinions  on  what  constituted  an  appropriate  level  of  achievement  in  terms  of  tasks 
(Section  4.2.2. 1).  Correspondingly,  there  was  a  tendency  for  some  individuals  to  complete  work 
and  to  delve  into  details  (Section  5. 1.2.4)  versus  advancing  it  only  to  a  certain  level. 
Consequently,  members  sometimes  expressed  a  degree  of  impatience  regarding  such  variable 
behaviour.  Impatience  was  also  noted  in  their  reaction  to  the  CEP  documentation  and  CEE 
availability /configuration,  each  of  which  impacted  the  team’s  work  practices. 

It  is  also  important  to  note  that  some  of  the  novelty-related  issues  are  endemic  to  the  early 
application  of  a  new  process  by  a  new  team  using  a  new  environment.  That  is,  some  of  the 
difficulties  encountered  by  the  CET  are  not  CapDEM-specific  but  would  be  common  in  any 
similar  situation.  Therefore,  these  kinds  of  issues  should  not  be  misattributed  and  result  in  a 


14  Note  that  these  attributions  are  not  intended  to  identify  or  be  critical  of  any  particular  individual  on  the 
CET;  rather,  they  are  to  highlight  common  personal  attributes  and  the  potential  to  result  in  unintended 
consequences. 


DRDC  Ottawa  TR  2011-044 


43 


falsely-based  critique.  Maintaining  such  separation  is  sometimes  difficult,  but  it  is  key  in  being 
able  to  properly  address  instance-specific  problems  and  to  distinguish  them  from  those  that  are 
more  generic. 

For  example,  while  the  specific  issues  the  CET  had  with  documentation  may  have  been  unique  to 
the  exercise,  issues  of  documentation  quality  and  consistency  are  common  to  any  new  and 
evolving  effort,  not  just  CEP.  While  it  is  important  to  highlight  relevant  particularities  for  the 
exercise,  such  as  the  issue  of  co-evolution  of  documentation  and  process,  the  key  observation  is 
its  degree  of  impact  on  the  team.  For  instance,  given  the  difficulty  a  relatively  small  team  had 
with  the  documentation  issues,  it  is  likely  that  such  issues  would  be  even  more  disruptive  in  a 
full-size  team?  Or  conversely,  would  the  proper  completion  of  all  roles  offer  a  degree  of  clarity 
such  that  documentation  issues  would  not  have  been  as  impactful?  Similarly,  the  rationale  for 
increased  coaching  is  not  unique  to  the  CEP,  but  indicative  of  its  level  of  maturity  and  a  lack  of 
familiarity  by  new  practitioners. 

Similar  considerations  include:  CBP  linkage,  CEP  understanding  and  process  assessment. 

•  CBP  linkage.  Throughout  the  exercise,  the  CET  spent  a  considerable  effort  trying  to  figure 
how  “it  all  fit  together”.  To  some  extent,  the  team  did  not  seem  confident  that  linkage  to 
appropriate  departmental  processes  had  been  considered  in  its  development  (“To  be  relevant, 
the  CEP  must  keep  pace  with  the  entire  capability  force  development  process  as  they  both 
continue  to  evolve  -  it  has  not.  Most  of  the  discussions  we’ve  had  with  the  capability 
planners  has  revealed  at  least  a  minor  inconsistency  or  deviation  and  would  therefore 
support  this  claim”).  Such  concern  and  commentary  belies  the  reality  of  the  process’  actual 
development  and  is  more  commensurate  with  issues  universal  to  product  readiness.  For 
example,  consider  any  given  software  development  effort:  the  testing  and  release  of  the 
product  are  based  on  ‘frozen’  versions  of  the  code  base  (i.e.,  business  rules)  despite  ongoing 
flux  towards  subsequent  releases  (i.e.,  as  the  business  rules  continue  to  evolve).  There  is  a 
natural  balance  between  practicality  and  agility  that  will  always  exist  in  such  a  context; 
therefore,  the  key  in  terms  of  CBP  linkage  and  the  CEP  is  find  that  point  of  balance.  It  is  not 
realistic  to  expect  multiple  moving  targets  (i.e.,  evolving  processes)  to  be  completely 
correlated,  particularly  when  they  are  person-based  (vs.  machine-based)  processes,  and 
therefore  subject  to  practitioner  variability  (e.g.,  familiarity,  competence,  etc.).  Furthermore, 
the  expectation  of  such  tight  coupling  is  not  amenable  to  evaluation  efforts,  reflecting  the 
issue  of  experimental  vs.  operational  focus  (see  ‘Confluence  of  initiatives’  under 
Section  5.1.1).  While  clearer  expectations  on  this  issue  are  expected  as  the  process  matures 
and  the  number  of  experienced  practitioners  increases,  it  is  recommended  that  future  CEP 
versions  formalize  more  easily  identifiable  linkages  (both  to  alleviate  these  concerns  as  well 
as  their  perception).  Furthermore,  as  the  development  and  utilization  of  the  process  become 
under  the  purview  of  a  single  organization  (e.g.,  CFD),  synchronization  will  undoubtedly 
become  more  straightforward  (than  was  possible  across  organizations,  as  in  CapDEM  and 
Exercise  Gamma). 

•  CEP  understanding.  Aside  from  any  complexities  inherent  in  SoS  engineering,  two 
challenges  overarching  CEP  understanding  were  its  newness  and  the  related  lack  of 
(practitioner)  expertise.  As  a  result  of  these  challenges,  the  CEP  was  arguably  too  easily  the 
subject  of  influence  from  ancillary  initiatives,  particularly  in  the  context  of  competing 
priorities  (Section  5.1.1).  That  is,  given  such  a  flurry  of  activity,  there  is  a  tendency  to 
question  that  which  is  new  and  unproven  (i.e.,  the  CEP),  rather  than  try  to  identify  the  actual 
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source  (such  as  an  existing  process).  The  result  is  a  predisposition  to  hastily  evaluate  efforts 
on  surface  considerations  (i.e.,  Tow  hanging  fruit’)  instead  of  a  more  thorough  deliberation 
of  why  things  were  as  they  were  (such  as  process  interfaces  or  the  suitability  of  other 
(possibly  existing)  processes).  For  example,  consider  the  comment  that  “[the]  keyword  is 
artefacts;  a  lot  are  not  what  VCDS  wants  to  see”.  Indeed,  this  statement  would  suggest  a 
lack  of  utility  in  terms  of  process  outputs.  However,  such  commentary  presumes  that  all 
outputs  from  the  process  are  directed  at  the  same  level  of  management  (i.e.,  decision).  In 
particular,  it  fails  to  realize  that  (fewer)  high-level  constructs  may  legitimately  benefit  from  a 
larger  number  of  low-level  constructs.  That  is,  the  process  needs  to  generate  outputs  that  are 
for  internal  use  and  analysis,  not  just  those  which  are  visible  to  the  final  management  level. 
This  issue  is  evident  between  the  exercise’s  various  governance  documents  (e.g.,  PLA  and 
CEP  initiative),  and  illustrates  the  cultural  predisposition  to  not  always  recognize  the  value 
of  intermediary  products  and  their  relevance  to  higher-level  outcomes.  Indeed,  it  is  this 
same  cultural  issue  that  impacts  the  ability  to  appreciate  the  iterative  nature  of  processes  like 
the  CEP  (as  discussed  under  ‘Iterative  progression’  in  Section  5. 1.2.1).  Previous  topics, 
such  as  the  capability  gap  and  CET  roles,  are  also  examples  of  issues  that  needed  to  be  re¬ 
evaluated  as  the  exercise  progressed. 

•  Process  assessment.  The  assessment  of  the  process  was  difficult  for  a  number  of  reasons. 
First,  the  CET  could  not  fully  appreciate  the  CEP  by  virtue  of  doing  it  in  a  condensed  (and 
skewed)  manner.  For  Exercise  Gamma,  the  CET  stated  they  would  have  preferred  a 
combined  process  rather  than  a  second  iteration  (because  they  were  so  late,  they  aimed  for  a 
lesser  amount  of  work);  however,  such  a  context  will  not  necessarily  be  the  case  when  the 
application  of  the  process  is  more  established  and  there  is  increased  familiarity  and  comfort 
in  terms  of  experienced  CETs.  Second,  there  was  a  natural  variance  in  the  conduct  of  the 
process  due  to  the  process  specification  being  more  descriptive  than  prescriptive.  However, 
the  amount  of  variance  in  Exercise  Gamma  was  regarded  as  atypical  of  that  going  forward, 
assuming  increased  process  maturity  and  practitioner  experience.  In  that  regard,  however,  it 
will  be  important  to  limit  any  workflow  customization,  so  as  to  prevent  unnecessary 
divergence  from  recommended  practices.  That  is,  it  will  be  important  to  ensure  that 
practitioners  know  and  utilize  ‘the  basics’  first.  Third,  the  CEP  must  be  applied  in  such  a 
way  as  to  enable  its  objectivity  and  traceability;  unfortunately,  its  application  in  Exercise 
Gamma  did  not  do  so  (“[The]  CEP  gives  objectivity;  the  way  we  did  it  was  not  objective  and 
lost  traceability  -  [there  is  a]  need  to  keep  objectivity  in  terms  of  [the]  process”).  Finally, 
there  is  a  need  to  appropriately  demarcate  between  issues.  The  classic  example  that  was 
experienced  in  the  exercise  was  keeping  process  understanding  and  domain  knowledge 
linked  but  separate.  During  the  initial  application  and  learning  of  a  new  process,  it  is  often 
difficult  to  disambiguate  between  new  knowledge  and  new  process.  While  they  will  seem 
the  same,  by  definition  they  are  not  -  such  that  knowledge  is  effectively  input  to  the  process 
and  the  process  is  independent  of  that  input.  However,  in  situations  when  ‘everything  is 
new’,  it  can  be  difficult  to  separate  them.  As  a  result,  it  can  be  difficult  to  discern  those 
which  are  problems  in  terms  of  domain  knowledge  versus  those  which  are  problems  in  terms 
of  the  process.  Indeed,  this  general  problem  does  exist  outside  the  CEP;  however,  it  is  key 
to  ‘keep  things  straight’  in  order  to  properly  assess  the  process  and  not  to  misattribute 
domain  knowledge  issues  as  process  failings. 
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5.1.3  Materiel 


Based  on  the  results  presented  in  Section  4.2.3,  the  following  discussion  provides  additional 
commentary  and  analysis  on  issues  pertaining  to  the  Materiel  axis.  Axis-relevant  artificialities 
are  presented  followed  by  considerations  along  the  following  themes: 

•  Understanding  and  usability 

•  Functional  grouping  and  alignment 

•  Information  and  knowledge 

•  Logistics 

•  Personnel 

•  Technology,  facilities  and  infrastructure 

In  terms  of  CEE  application,  various  artificialities  relating  to  the  effort  being  an  exercise  (rather 
than  an  actual  operation)  should  be  noted.  These  include: 

•  Novelty.  Combined  with  the  security  requirements  of  Exercise  Gamma,  the  diverse  and 
developmental  nature  of  the  classified  CEE  made  it  difficult  to  delineate  systemic  challenges 
from  those  which  were  transient.  That  is,  it  was  sometimes  unclear  whether  certain  issues 
(e.g.,  performance,  interoperability,  etc.)  were  specific  to  the  way  the  CET  was  attempting  to 
utilize  a  particular  tool,  that  tool’s  configuration,  the  specific  underlying  infrastructure  being 
used,  an  inherent  technological  limitation,  or  a  regulatory  (policy)  consequence.  As  a  result, 
the  team’s  perception  of  the  technology  (i.e.,  the  tools,  their  availability,  dependability  and 
performance)  was  adversely  impacted  (i.e.,  skewed)  based  on  the  idiosyncrasies  of  a 
developmental  prototype. 

•  Context  and  Readiness.  The  development  of  the  classified  CEE  paralleled  the  ‘confluence 
of  initiatives’  and  ‘meta-influences’  that  impacted  both  the  CET  and  the  CEP.  Specifically, 
CEE  development  and  management  were  under  the  auspices  of  DRDC  CORA  (as  opposed 
to  the  CapDEM  TDP),  which  resulted  in  sometimes  competing  priorities.  The  end  result 
was  that  its  development  timeline  was  driven  somewhat  independently  of  the  actual  exercise. 
Specifically,  the  status  of  CEE  afforded  a  limited  amount  of  flexibility  in  terms  of 
configuration,  availability  and  accessibility  to  the  environment  and  its  associated  facilities 
(e.g.,  the  classified  ACCESS  Lab13). 

A  lack  of  familiarity  with  the  CEE  and  its  components  combined  with  deficient  and 
ineffective  training  further  contributed  to  insufficient  readiness.  Information  validity, 
process  integration,  as  well  as  specific  functional  and  configuration  issues  (as  detailed 
below)  were  also  overriding  concerns.  While  some  of  these  concerns  were  unique  to  the 
exercise  context,  they  often  preoccupied  CET  members  in  a  manner  unlikely  to  be  found  in 
an  actual  operational  situation  (in  which  more  time-tested  environments  would  be  used). 

As  with  the  CEP,  a  number  of  issues  outside  the  exercise’s  sphere  of  influence  had 
significant  impact  on  the  use  of  the  CEE.  Examples  of  such  ‘meta-influences’  include  the 


15  The  classified  ACCESS  Lab  initiative  began  as  part  of  the  CapDEM  TDP  but  quickly  transitioned  to  an 
independent  effort  managed  primarily  by  DRDC  CORA.  Consequently,  the  lab  was  and  still  is  commonly 
referred  to  as  ‘ACCESS  CORA’. 
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operational  constraints  and  configuration  rules  associated  with  various  departmental 
resources;  a  case  in  point  being  network  policies  and  security  regulations.  Specifically,  the 
ability  to  address  such  issues  is  more  dependent  on  the  use  and  availability  of  well-supported 
operational  infrastructures  and  suitable  technical  personnel  than  it  is  on  the  approach  itself. 
Consequently,  the  successful  realization  of  the  CEE  depended  on  critical  factors  outside  of 
its  control.  As  Exercise  Gamma  was  the  first  ‘significant’  effort  to  be  executed  on  the  new 
classified  CEE,  a  variety  of  technical,  logistical,  training  and  operational  issues  came  to  the 
fore.  Further,  the  ongoing  evolution  of  CEE  design  and  architecture  as  part  of  exploring 
how  to  better  meet  the  broad  process  and  personnel  requirements  made  it  difficult  to 
converge  towards  a  ‘definitive’  CEE.  That  is,  because  the  implementation  of  the  CEE 
necessarily  changed  as  a  matter  of  course  throughout  CapDEM  as  well  as  between  the 
evaluation  exercises,  there  was  a  lack  of  confidence  and  certainty  in  terms  of  its  robustness. 
Based  on  the  maxim  that  “the  devil  is  in  the  details”,  various  lessons  (from  technical, 
operational  to  logistical)  were  learned  from  the  considerable  and  often  unforeseen  effects  of 
seemingly  minor  changes  in  terms  of  implementation  details.  Hence,  the  challenge  is  to 
appropriately  reflect  such  a  cause/effect  hierarchy  in  a  manner  useful  in  guiding  future  CEE 
design  and  development. 

5.1 .3.1  Understanding  and  Usability 

The  tightly  coupled  themes  of  understanding  and  usability  underlie  many  of  the  issues  that 
affected  the  CEE  during  Exercise  Gamma.  In  particular,  a  lack  of  understanding  of  how  to  use 
particular  tools  as  well  as  how  to  use  them  in  conjunction  with  each  other  exacerbated  the  level  of 
difficulty  experienced  (i.e.,  usability).  In  general,  the  utility  of  the  CEE  was  not  appreciated  nor 
capitalized  upon  (i.e.,  taken  advantage  of)  by  the  CET.  The  CET  did  not  effectively  or  efficiently 
utilize  many  elements  of  the  CEE  and  exhibited  the  classic  ‘training  vs.  doing’  divide.  For 
example,  while  the  team  expressed  a  desire  for  more  and  better  training  of  the  CEE,  doing  so  was 
afforded  little  time  and  availability  by  the  CET.  Similarly,  there  was  often  concern  over  whether 
the  tool  suite  was  sufficient  or  modifiable  enough  (i.e.,  customizable)  to  do  the  work  at  hand 
while  not  yet  having  a  sufficient  sense  of  what  tool  could  do;  that  is,  they  often  worried  about 
whether  the  CEE  was  capable  of  a  function  and  who  would  be  responsible  for  it,  rather  than 
investigate  how  the  tools  could  have  worked  together  to  meet  their  needs.  This  issue  is  an 
example  of  how  the  team’s  work  methodology  (e.g.,  curiosity  vs.  instructive)  was  incompatible 
with  the  non-operational  readiness  of  the  CEE.  Combined  with  issues  of  workload  (e.g.,  time), 
the  issues  of  knowledge  and  awareness  became  pivotal. 

As  part  of  working  together,  all  members  of  the  CET  need  to  be  at  ease  with  entire  tool  suite  to 
avoid  potential  communication  problems  through  better  understanding  of  other  participants  and 
each  other’s  contributions  to  the  effort.  That  is,  the  CET  requires  a  global  sense  of  CEE 
capabilities  through  a  basic  level  of  knowledge,  such  that  people  can  access  information  for  their 
own  puiposes.  It  is  not  intended  that  everyone  should  have  the  level  of  knowledge  required  to 
use  any  given  tool  to  process  information.  Subject  matter  experts  and/or  specialized  users  are  still 
warranted,  but  as  part  of  forming  a  cohesive  holistic  team,  knowing  ‘the  language’  of  the  work 
environment  is  a  necessary  and  important  part  of  enabling  users’  trust  in  the  technology  as  well  as 
other  users  and  their  use  of  the  CEE. 

The  above  discussion  alludes  to  the  general  issue  of  connectedness  both  amongst  CET  members 
and  between  CET  members  and  their  work  environment  (i.e.,  the  CEE).  In  situations  where 
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“everything  is  new”,  be  it  team  members,  the  work  processes  they  are  applying,  or  the  technology 
they  are  using,  it  is  important  to  ‘ground’  people  with  a  sense  of  certainty.  Given  the  objective 
and  pervasive  nature  of  technology,  it  is  often  intended  to  address  this  need;  however,  in  Exercise 
Gamma,  this  was  not  generally  the  case.  Due  to  the  novelty  and  diversity  of  the  CEE,  many 
members  of  the  CET  had  no  prior  exposure  to  the  tools  being  used;  as  a  result,  their  level  of 
comfort  was  less  than  optimal.  Consequently,  there  was  sparse  usage  of  many  CEE  components 
by  such  individuals,  and  various  tools  (such  as  the  Livelink  portal/repository)  were  used  in  less 
than  an  ideal  manner.  The  previously  mentioned  difficulties  with  training  and  the  influence  of 
demographics  further  exacerbated  the  situation  in  which  surprising  approaches  to  tool  use  and 
associated  requests  for  support  were  problematic  for  the  Evaluation  Team  to  address  in  a  timely 
fashion.  As  a  result,  satisfaction  across  the  whole  of  the  CEE  was  generally  lessened. 

An  unfortunate  consequence  of  the  above  was  a  lack  of  cohesiveness  and  ownership  with  respect 
to  the  use  of  the  technology.  Specifically,  there  was  an  expressed  desire  to  have  certain  tools 
actually  used  on  behalf  of  the  CET  (i.e.,  have  an  operator  provided).  In  fact,  depending  on  the 
tool  and/or  the  type  of  task  to  be  done,  there  sometimes  seemed  to  be  a  disdain  towards  tool  use, 
particularly  in  the  sense  of  preparatory  (e.g.,  foundational,  organizational)  steps  which  were 
seemingly  regarded  as  an  administrative  function  instead  of  a  relevant  aspect  of  their  work. 
While  some  CET  members  were  strong  advocates  of  specific  tools  and  their  potential  within 
application  of  the  CEP,  there  sometimes  seemed  to  be  a  general  sense  of  apathy  amongst  many 
members  in  terms  of  the  CEE  and  a  reduced  sense  of  responsibility  for  their  use  in  specific  work 
contexts  (for  example,  see  ‘data  model’  under  Section  5. 1.3. 3).  Going  forward,  a  much  more 
cohesive  approach  and  a  clearer  delineation  of  responsibilities  with  respect  to  technology  use, 
along  with  a  stronger  ability  to  assist  in  problematic  areas  will  be  required  (see  discussion  under 
sections  5. 1 .3.5,  5. 1 .3.6  and  5.2.2). 

Finally,  it  should  be  noted  that  the  sense  of  urgency  incurred  during  the  latter  weeks  of  the 
exercise  (see  Section  5. 1.2.1)  provided  a  different  experience  for  the  CET  in  terms  of  the  CEE 
(vs.  the  CEP).  Unlike  the  CEP,  in  which  the  CET  made  stronger  attempts  to  execute  the  process 
during  its  schedule  extension,  tool  usage  was  not  approached  in  the  same  manner.  Specifically, 
the  desire  to  expedite  their  work  and  not  ‘waste  time’  resulted  in  the  CET  following  ‘the  path  of 
least  resistance’  with  respect  to  tool  usage.  Hence,  some  elements  of  the  CEE  had  reduced  usage 
(as  there  was  “no  time”)  or  some  elements  were  used  in  a  limited  or  unintended  (sometimes 
incorrect)  manner.  Thus  while  the  sense  of  urgency  arguably  helped  focus  process  issues,  it 
tended  more  often  than  not  to  inhibit  and/or  hinder  materiel  application. 

5.1 .3.2  Functional  Grouping  and  Alignment 

As  mentioned  in  the  introduction  to  Section  5.1.3,  while  there  was  the  notion  of  a  ‘standard 
CapDEM  toolset’  for  purposes  of  conducting  the  TDP,  the  ongoing  evolution  of  the  CEE 
throughout  CapDEM  made  it  difficult  to  converge  towards  a  ‘definitive’  CEE16.  Arguably,  while 
it  could  be  useful  (in  some  ways)  to  direct  that  specific  tools  and/or  functions  be  used  at  particular 
points  in  the  process,  doing  so  is  inappropriate  for  several  reasons: 


16  The  notion  of  ‘definitive’  denotes  the  specification  of  the  CEE  at  the  granularity  of  specific  and 
individual  tools  (e.g.,  product  name,  vendor,  version,  etc.). 
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•  Evolution.  Excessively  fine  granularity  in  terms  of  process  and  tool  interaction  makes 
application  of  the  process  rigid  and  the  technology  fragile;  that  is,  because  both  axes  will 
evolve  over  time,  overly  tight  integration  would  require  an  untenable  level  of  maintenance  of 
both  axes  to  ensure  their  continued  alignment.  In  addition  to  the  commercial  and  political 
sensitivities  of  specifying  particular  hardware  and  software,  overt  dependencies  between 
process  steps  and  specific  tool  features/functions  can  easily  become  out-of-date,  resulting  in 
confusing,  incorrect  and  potentially  counter-productive  or  inoperative  linkages. 

•  Adaptability.  Excessively  fine  granularity  in  terms  of  process  and  tool  interaction  also  does 
not  readily  facilitate  environment  adaptation.  As  each  new  instantiation  (i.e.,  instance)  of 
the  CapDEM  approach  is  realized  based  on  its  own  problem  specification  and  with  its  own 
CET,  there  is  the  need  to  be  able  to  adapt  the  supporting  CEE  environment  based  on 
instance-specific  preferences  and  requirements.  Without  doubt,  the  commonalities  in  terms 
of  interface  and  organization  between  various  CEEs  were  highly  beneficial  and  appreciated 
by  its  users.  However,  the  open-ended  potential  of  requirements  resulting  from  a  broad  user 
base,  diverse  usage  contexts  and  situation-specific  logistics  (ranging  from  government 
procurement  practices  to  technical  infrastructure  considerations)  did  limit  the  ability  and 
utility  of  recommending  specific  applications. 

•  Philosophy.  Excessively  fine  granularity  in  terms  of  process  and  tool  interaction  also  goes 
against  the  non-prescriptive  philosophy  taken  both  with  CEP  and  CEE  usage.  As  with  the 
process,  the  intent  (and  hope)  was  not  to  prescribe  technology  but  enable  informed,  creative 
and  evolutionary  solution  development  by  the  CET  through  the  provision  of  classes  of  tools 
to  enable  and  facilitate  the  various  engineering  and  analysis  tasks.  That  is,  an 
inappropriately  tight  coupling  between  process  and  materiel  axes  does  not  readily  allow  for 
individual  variations  in  work  practices  and  novel  (i.e.,  creative)  approaches  therein.  A 
balanced  approach  which  provides  alignment  between  technology  and  specific  CET  roles 
therefore  offers  educated  users  a  means  to  capitalize  on  best  practices  and  individual 
expertise. 

Accordingly,  the  approach  taken  was  to  identify  functional  groupings,  validate  their  use  and 
document  their  issues  and  evolution  throughout  execution  of  the  TDP.  Within  Exercise  Gamma, 
for  example,  an  option  analysis  tool  proposed  by  one  of  the  CET  members  was  added  to  the  CEE 
to  augment  its  existing  functionalities.  Other  changes,  such  as  those  necessitated  by  the 
exercise’s  security  classification,  were  evolutions  due  to  changing  usage  requirements. 
Furthermore,  while  CET  feedback  highlighted  the  need  to  add  risk  and  costing  tools,  the  need  for 
support  in  those  functional  categories  had  been  previously  anticipated;  and  in  the  case  of  costing, 
a  specific  tool  had  been  provided  to  earlier  CETs.  However,  that  tool  was  later  found  to  be 
unsuitable  and  subsequently  removed  from  the  CEE.  Nonetheless,  the  confirmation  of  need 
highlights  the  value  in  identifying  various  possible  evolutionary  paths  for  the  CEE,  including  the 
need  to  address  functional  areas  that  are  not  well-served  by  currently  available  tools  and 
technologies  (i.e.,  promote  their  development  either  within  industry  or  as  research  efforts). 

Consequently,  a  more  utilitarian  approach  was  to  formulate  a  CEE  reference  architecture,  defined 
in  a  reasonably  ‘tool-agnostic’  fashion.  Note,  however,  that  such  an  approach  can  still  come  with 
significant  challenges  in  terms  of  implementation  (see  Section  5. 1.3. 6).  Additionally,  while  a 
functional  approach  is  useful,  it  can  be  difficult  to  discern  and  convey  the  idiosyncrasies  of 
general  usage  versus  the  means  to  realize  a  particular  effect.  An  example  from  Exercise  Gamma 
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would  be  the  use  of  Livelink  for  both  general  file  management  as  well  as  trying  to  make  it  serve 
as  a  data/model  repository  (see  Section  5. 1.3.3). 

5.1 .3.3  Information  and  Knowledge 

As  with  the  issues  of  understanding  and  usability,  two  broad  concerns  in  terms  of  information  and 
knowledge17  management  also  underpin  CEE  utilization:  (1)  information  organization, 

representation  and  access;  and  (2)  classification  of  information  and  the  facilities  (e.g.,  transport, 
communication,  storage,  separation,  procedures,  etc.). 

Organization,  representation  and  access.  The  organization  of  the  information  utilized  within 
Exercise  Gamma,  including  its  representation  and  access  (availability),  was  of  considerable 
concern  by  the  CET.  Part  of  this  concern  was  in  terms  of  the  knowledge  portal,  realized  via  a 
content  management  system  (i.e.,  Livelink),  for  use  as  a  central  repository  and  collaboration  hub. 

Unfortunately,  the  reality  of  portal  use  was  not  as  anticipated.  Specifically,  it  was  used  less  and 
in  a  less  than  advantageous  way  both  in  terms  of  information  storage  and  collaboration.  Indeed, 
portal  use  and  the  CET’s  information  management  practices  were  impacted  by  a  number  of 
factors:  existence  and  awareness  of  infrastructural,  compatibility  and  performance  issues; 
personnel  and  training  issues,  including  understanding  (of  the  tool  and  information  management 
practices)  and  user  preferences  (e.g.,  favouritism  towards  other  tools).  There  was  significant 
reticence  by  members  in  terms  of  how  to  address  these  concerns;  specifically,  numerous  CET 
members  wanted  assistance  to  utilize  Livelink  for  tasks  such  as  creating  and  organizing 
documents,  including  the  maintenance  of  versions  and  relationships  between  them.  The  level  of 
assistance  ranged  from  an  on-call  coach  to  the  preference  of  ‘taskable’  support  personnel  to  do 
these  tasks  on  their  behalf.  This  request  was  not  expected  and  efforts  to  provide  the  desired  level 
of  support  could  not  be  realized  in  sufficient  time  to  be  useful.  Indeed,  it  was  anticipated  (by  both 
design  and  experience-to-date)  that  the  CET  itself  would  be  sufficiently  capable,  interested  and 
motivated  to  utilize  the  portal  and  organize  its  information  in  the  repository  by  itself.  However, 
this  was  not  the  case  and  to  some  degree,  such  an  expectation  was  both  resisted  and  derided. 

In  general,  the  approach  taken  was  more  akin  to  a  network  drive  than  for  collaboration  and 
content  (information)  management.  Accordingly,  the  use  of  configuration  management  (CM) 
was  an  issue.  Ideally,  the  focus  would  have  been  the  application  of  CM  principles  using  the 
provided  toolset;  however,  such  an  approach  was  problematic  given  the  existing  issues  of 
understanding  and  information  management.  Consequently,  a  more  conscious  effort  on  how  to 
utilize  CM  practices  within  the  CEE  needs  to  be  made. 

For  example,  consider  the  notion  of  an  ‘artefact-centric’  approach  to  the  CEP  (see  Section  5.1.2) 
and  the  desire  to  move  away  from  a  ‘document-oriented’  approach  (i.e.,  “...less  like  writing  a 
report...”).  In  such  a  case,  in  which  report  generators  would  automatically  compose  deliverables 
from  artefacts,  the  unit  of  consequence  is  an  ‘information  fragment’  such  that  deliverables  and 
tasks  refer  to  and  exploit  the  same  information  fragment  (i.e.,  artefact).  Doing  so  reduces 
duplication  and  assists  in  reducing  error  and  maintaining  currency  between  related  tasks  and 
deliverables.  Furthermore,  the  use  of  such  a  ‘composable’  approach  has  the  potential  to  be 


17  For  purposes  of  this  discussion,  the  terms  ‘information’  and  ‘knowledge’  are  generally  used 
interchangeably,  except  when  explicitly  differentiated  to  identify  specific  characteristics. 
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automated.  However,  such  an  orientation  has  a  significant  effect  on  tool  use  and  information 
structure.  In  particular,  an  artefact  is  at  a  higher  granularity  than  a  document  (e.g.,  a  paragraph, 
figure  or  table  which  together  comprise  a  document);  therefore,  an  appropriate  information  (data) 
model  must  exist  by  which  final  outputs  (e.g.,  final  reports,  deliverables)  can  be  specified  as 
functions  of  elements  of  the  information  model.  As  the  standard  COTS  tools  utilized  within  the 
CEE  function  at  a  ‘document’  level,  a  tighter  and  customized  approach  would  be  required  to 
reformulate  deliverables  as  composites  of  a  finer  granular  data  model.  In  terms  of  Livelink,  its 
nested  constructs  (i.e.,  compound  documents18  and  documents)  could  have  been  explored  as  a 
potential  way  to  do  so.  Indeed,  this  approach  was  proposed  by  the  Materiel  axis  advisor; 
however,  the  resources  necessary  to  realize  the  approach  were  not  available.  Going  forward, 
there  would  be  significant  benefit  to  providing  for  varying  granularity  and  composability  in  terms 
of  information  management.  Notably,  a  degree  of  CEE  customization  (e.g.,  connectivity  between 
architecture  and  documentation  tools)  and  tighter  integration  between  axes  would  be  required  to 
more  clearly  identify  which  role  and  technology  would  align  with  (i.e.,  be  responsible  for)  a  given 
artefact  and  the  relevant  part(s)  of  the  process. 

A  related  information  management  peculiarity  was  the  CET’s  request  for  the  ability  to  have 
“versions  of  relationships  between  ‘things’”.  The  standard  approach  to  CM  utilizes  versions  of 
‘things’  (such  as  documents)  with  known  relationships  between  them.  Accordingly,  in  the  case 
such  that  a  ‘thing’  or  the  nature  of  a  relationship  between  ‘things’  change,  a  new  ‘configuration’ 
results.  Within  Exercise  Gamma,  it  was  unclear  how  such  a  model  could  be  implemented  within 
the  CEE  (i.e.,  Livelink);  specifically,  it  would  have  required  more  expertise  and  time  than  was 
available  to  design  and  implement  any  probable  solution.  Further,  the  Evaluation  Team  (through 
the  Materiel  axis  advisor)  also  deemed  such  an  approach  as  inappropriate.  For  example,  consider 
two  possibilities  in  terms  of  realizing  a  ‘versioned  relationship’  within  a  configuration  managed 
informational  model:  (1)  link  a  version  of  ‘Document  A’  to  a  version  of  ‘Document  B’  to  a 
version  of  ‘Document  C’  (effectively  a  transitive  baseline  across  a  group  of  versioned 
documents);  or  (2)  create  a  version-able  object  which  itself  can  represents  different  kinds  of 
relationships  (as  an  instantiable  first-class  object)  combined  with  the  use  of  tagging  (such  as 
through  Livelink  classifications).  Aside  from  the  complexity  of  understanding  the  issue, 
addressing  it  was  deemed  a  ‘red  herring’  and  outside  the  scope  of  the  exercise  (that  is,  it  was  not 
required  in  order  to  do  the  work).  Rather,  the  issue  was  an  (interesting)  consequence  of  the  way 
the  CET  approached  their  information  management.  Going  forward,  the  linkages  and  impact 
between  information  modelling,  tool  outputs,  process  outputs  (e.g.,  deliverables)  and  their 
construction  relative  to  each  other  needs  more  consideration.  The  importance  of  training  and 
suitable  SMEs  in  terms  of  these  issues  also  needs  to  be  more  clearly  articulated,  so  as  to  help 
ensure  the  CET  maintains  focus  and  avoids  unnecessary  tangents.  Further,  there  needs  to  be 
clarity  in  terms  of  the  suitability  and  the  implications  of  using  the  same  or  different  technologies 
(tools)  for  both  general  file  management  and  for  a  structured  data/model  repository. 

In  terms  of  the  impact  of  information  management  practices  on  collaboration,  the  CET  generally 
employed  classic  editing  of  individual  documents  rather  than  collaborative  use  of  shared 
documents.  Indeed,  the  CET  could  have  used  the  CEE  functionalities  to  be  collaborative,  but 
they  would  have  had  to  make  an  explicit  effort  and  manage  documents  according  to  a  particular 
methodology.  Such  a  tendency  reflected  a  lack  of  experience  with  collaborative  systems  along 


18  A  ‘compound  document’  is  the  Livelink  nomenclature  for  one  composed  of  other  individual  documents. 
The  idea  is  akin  to  MS  Word’s  ‘master  document’  and  the  notion  of  ‘object  linking  and  embedding’  (OLE). 


DRDC  Ottawa  TR  2011-044 


51 


with  unreasonable  expectations  upon  the  collaborative  system;  indeed,  while  many  collaborative 
technologies  are  transparent,  the  logistics  of  sharing  and  the  potential  side  effects  on  the  materials 
being  edited  need  to  be  kept  in  mind.  Notably,  CEE  performance  (specifically  Livelink)  was  an 
issue  impacting  their  desire  to  use  these  features.  Consequently,  the  situation  was  an  example  of 
infrastructure  impacting  performance  impacting  choices  of  tool  feature/function  impacting  the 
overall  extent  of  collaborative  behaviour.  Specifically,  in  situations  such  that  certain  behaviours 
are  not  routine,  small  technical  issues  can  be  seen  to  have  unintended  consequences  in  which 
people  will  revert  back  to  ‘what  works’  and  ‘the  path  of  least  resistance’. 

Lastly,  the  collection  of  information  was  also  difficult  in  terms  of  input  from  organizational  levels 
above  the  CET.  Specifically,  there  was  a  desire  expressed  for  the  use  of  an  enterprise  architecture 
tool  to  assist  in  getting  and  organizing  system  attributes.  More  generally,  this  issue  speaks  to  the 
need  to  provide  technological  support  along  the  interface  between  the  CEP  and  other 
departmental  processes.  Going  forward,  clarity  as  to  relevant  organizations,  processes  and 
systems  will  need  to  be  provided  as  part  of  addressing  this  issue;  however,  for  puiposes  of  the 
evaluation  exercise,  this  concern  was  beyond  scope. 

Classification.  Information  classification  and  the  means  by  which  to  transport,  communicate  and 
process  said  information  was  an  overriding  concern  of  the  CET.  Of  particular  note  was  the  desire 
for  an  expeditious  means  to  migrate  from  an  unclassified  to  a  classified  CEE.  Both  of  these 
considerations  were  frustrated  by  the  prototypical  nature  of  the  classified  CEE,  such  that  there 
were  still  ‘growing  pains’  associated  with  its  finalization  and  initial  use.  Combined  with  the 
issues  of  information  management  (see  above),  logistics  (Section  5. 1.3.4)  and  infrastructure 
(Section  5. 1.3.6),  the  result  was  an  inconsistent  and  sporadically  unavailable  environment  which 
frustrated  both  the  CET  as  well  as  the  Evaluation  Team. 

Despite  the  legitimacy  of  the  above  concerns,  classification  was  sometimes  a  pretext  for  not 
resolving  (and  even  avoiding)  certain  issues;  that  is,  it  was  easy  to  blame  the  environment.  For 
example,  the  CET  said  the  environment  was  ‘not  ready’  to  deal  with  classified  material. 
However,  given  the  environment  was  certified  to  Level  II,  such  readiness  was  characteristically 
more  procedural  and  knowledge-related.  That  is,  since  the  exercise  was  the  first  use  of  the 
technology  (both  as  an  environment  and  by  many  individual  CET  members),  knowing  what  and 
how  to  do  things  were  mutually  confounding  factors.  Rather,  the  bigger  difficulty  was  the  ability 
to  be  collaborative  due  to  facility  issues  (e.g.,  lack  of  classified  work  places,  network  availability, 
etc.)  combined  with  the  knowledge  issue  of  how  to  access,  process  and  exchange  information. 
Indeed,  the  collection  of  information  from  other  sources,  including  SMEs  and  other  classified 
environments,  was  difficult  both  because  of  such  technical  limitations  but  also  due  to  a  lack  of 
procedural  experience.  Furthermore,  the  use  of  the  ‘Canadian  Eyes  Only’  (CEO)  caveat  stifled 
collaborative  efforts  due  to  most  departmental  classified  systems  being  CAN/US,  not  CEO. 
Thus,  the  problem  of  classification  (and  caveat)  must  be  addressed  and  educated  at  a  higher 
organizational  level  than  possible  within  a  single  instantiation  of  the  CapDEM  approach;  that  is, 
the  CEE  must  fit  relative  to  the  broader  departmental  capabilities  and  procedures  as  part  of 
facilitating  institutionalization  and  growing  the  interface  between  resources  considered  part  of  the 
CapDEM  approach  and  other  relevant  departmental  actors. 
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5.1 .3.4  Logistics 


The  above  concerns  regarding  facilities  (e.g.,  lack  of  classified  work  places,  network  availability, 
etc.)  speak  to  the  larger  issue  of  exercise  logistics  and  the  logistics  of  applying  the  CapDEM 
approach.  While  these  two  issues  intersect,  it  is  important  to  realize  they  were  not  the  same  and 
will  naturally  differentiate  going  forward. 

Logistics,  as  typified  by  the  availability  and  access  to  facilities  and  technical  support,  proved  to 
be  a  preoccupation  for  the  CET  within  Exercise  Gamma.  Specific  issues  such  as  team  mobility 
and  a  desire  for  co-location  were  encountered.  There  was  also  the  desire  to  be  integrated  into 
(i.e.,  connected  to)  the  classified  CEE  from  individual  member’s  offices  (workstations).  Notably, 
the  connection  to  the  classified  CEE  outside  its  physical  boundaries  (consisting  of  the  classified 
ACCESS  Lab  and  adjoining  server  and  storage  rooms),  would  have  required  a  complete  virtual 
environment  based  on  external  connectivity  to  the  dedicated  CEE  facilities. 

Interestingly,  the  requirement  for  a  complete  virtual  environment  was  somewhat  at  odds  with  the 
team’s  opposition  to  distributed  meetings.  Indeed,  CET  desire  for  co-location  would  limit  the 
potential  value  of  employing  a  distributed  environment.  However,  it  would  seem  that  the  primary 
motivation  behind  the  virtual  environment  (‘virtual  facility’)  was  to  enable  the  use  of  the 
classified  CEE  from  individual  offices  in  order  to  avoid  the  need  to  use  a  dedicated  classified 
facility,  such  as  the  classified  ACCESS  Lab.  As  the  use  of  specialized  facilities  for  classified 
meetings  has  been  the  norm  and  there  was  no  precedence  for  such  an  office-oriented  approach,  it 
seemed  that  such  a  request  was  based  primarily  on  workplace  comfort  and  preference  (possibly 
indolence). 

It  should  be  noted  that  many  of  these  concerns  are  outside  the  scope  the  exercise  and  while  they 
are  relevant  to  individuals  within  the  CET  in  terms  of  their  experience  in  applying  the  CapDEM 
approach,  they  are  less  affected  by  CapDEM-specific  issues  and  more  about  general  resource  and 
personnel  issues.  An  example  is  that  commonplace  office  logistics  (e.g.,  moving  offices, 
transferring  between  buildings)  were  not  an  exercise-level  responsibility.  Rather,  the  exercise 
management  attempted  to  mitigate  the  impact  of  such  issues  on  the  CET  members;  however,  the 
provision  of  secure  computing  at  individual  workstations/offices  was  an  issue  broader  than 
CapDEM.  Specifically,  it  was  not  possible  (nor  realistic)  to  address  all  the  logistics  (technical 
and  security)  that  resulted  (e.g.,  ensuring  network  runs  to  new  office  spaces  and  providing  for 
structural  constraints  to  enable  secure  office  processing  (including  window  coverings  and 
controlled  physical  access)).  While  the  exercise  provided  an  example  context  and  additional 
impetus  to  provide  for  such  a  capability,  it  would  be  extremely  unlikely  to  be  a  sufficient 
requirement  and  could  not  be  the  sole  source  of  resources  to  address  such  needs. 

Within  Exercise  Gamma,  logistical  issues  detracted  from  CET  focus  and  CEE  application.  Going 
forward,  clearly  identified  and  reliable  logistic  support  will  be  required  to  ensure  both  of  these 
needs  are  met.  This  support  includes  the  need  to  ensure  environmental  and  infrastructural 
readiness  in  a  more  reliable  and  timely  manner,  as  part  of  not  undervaluing  the  importance  of 
these  issues  to  the  CET  and  to  the  successful  use  of  the  CEE.  Conversely,  the  difficulty 
associated  with  achieving  such  a  reality  should  not  be  underestimated  given  the  various  issues 
outlined  in  Section  5. 1.3.6. 
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Naturally,  with  increased  experience  will  come  fewer  problems  in  terms  of  how  to  access,  process 
and  exchange  information.  The  collection  of  information  from  other  sources,  including  SMEs 
and  other  classified  environments  will  change  over  time  as  the  process  becomes  more  embedded 
in  departmental  practice,  and  both  experience  and  benefits  accrue,  promoting  awareness  and 
lessening  concern  through  familiarity  and  acceptance  by  the  user  community.  Further,  the  design 
and  capabilities  of  facilities  will  continue  to  evolve  based  on  contemporary  work  practices. 

Notably,  a  benefit  resulting  from  CET  interest  in  using  the  classified  CEE  at  external  offices 
(workstations)  was  the  highlighting  of  its  mutually  exclusive  configuration  constraint19.  Indeed, 
the  notion  of  realizing  a  concurrent  yet  dissimilar  virtual  environment  configuration  to  that  active 
in  the  lab  is  a  lesson  learned.  Dual  internal/external  configurations  were  not  part  of  the  original 
design  specification;  hence  Configuration  A  being  active  in  the  lab  implies  Configuration  A  will 
be  active  on  the  external  virtual  environment,  such  that  a  different  Configuration  B  would  not  be 
useable  at  that  time.  This  constraint  could  result  in  potential  conflicts  between  internal  and 
external  CEE  users.  Going  forward,  should  security  issues  be  addressed  such  that  classified  CEE 
access  external  to  the  classified  ACCESS  Lab  become  viable,  further  examination  as  to  the 
support  of  different  internal  and  external  configurations  will  be  required.  Indeed,  whether  or  not 
multiple  external  configurations  could  be  made  available  at  the  same  time  would  also  require 
further  study  and  engineering.  Presumably,  the  value  of  such  functionality  would  markedly 
increase  should  CEE  use  become  commonplace  given  the  institutionalization  of  the  CapDEM 
approach,  in  part  based  on  the  frequency  and  relative  consequence  of  such  requests.  Notably, 
however,  based  on  previous  development  experience  with  the  classified  CEE  and  ACCESS  Lab, 
achieving  such  modifications  would  not  be  trivial. 

5.1 .3.5  Personnel 

A  critical  factor  impacting  CEE  application  is  personnel  availability,  training  and  integration. 
While  the  technical  considerations  of  the  CEE  are  obviously  a  critical  enabler,  those  using  and 
supporting  the  CEE  are  the  lynchpin  underpinning  its  successful  application.  In  particular,  the 
provision  of  quality  technology  without  appropriate,  sufficient  and  effective  linkage  to  personnel 
and  process  axes  will  result  in  less  than  optimal  benefit.  Thus,  ensuring  the  appropriately  skilled 
personnel  are  on  the  CET,  either  as  core  members  or  SMEs,  is  a  key  facet  of  realizing  a  beneficial 
application  of  the  CEE. 

Within  Exercise  Gamma,  the  CET  did  not  utilize  the  functionality  of  certain  tools  in  a  suitable 
manner.  While  there  was  the  intention  to  allow  for  exploration,  novelty  and  creativity  in  the  use 
of  the  CEE,  the  Materiel  Advisor  (Evaluation  Team)  attempted  to  broadly  steer  the  CET  towards 
certain  flexible  but  utilitarian  approaches.  However,  the  degree  to  which  assistance  could  be 
provided  was  outpaced  by  the  needs  of  the  CET,  resulting  in  members  using  ad  hoc  approaches 
that  sometimes  did  not  provide  a  sustainable  solution.  Sometimes  these  approaches  became 
sufficiently  ingrained  that  changing  them  would  have  been  problematic  given  the  time  and 
resources  available.  These  circumstances  typically  aligned  with  a  lack  of  appropriate  knowledge 
in  terms  of  individual  technologies,  general  organizational  and  information  (knowledge) 


19  A  configuration  in  this  context  refers  to  the  combination  of  select  active  networks  (e.g.,  network 
enclaves)  at  any  given  moment.  Specifically,  due  to  security  restrictions,  only  certain  networks  can  be 
active  within  the  environment  at  the  same  time  so  that  likelihood  of  cross-contamination  between  them  is 
minimized. 
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management  practices,  along  with  unrealistic  expectations.  The  epitome  of  this  situation  was  the 
use  of  the  Livelink  content  management  system  as  an  information  and  knowledge  management 
tool.  Specifically,  there  was  a  significant  disconnect  between  how  the  CET  envisioned  their 
information,  how  it  should  be  organized  and  represented  both  conceptually  and  within  Livelink, 
as  well  as  the  knowledge,  ability,  time  and  resources  available  to  the  CET  (and  the  Evaluation 
Team). 

At  the  same  time,  there  was  an  apparent  lack  of  interest  by  the  CET  to  directly  address  these 
issues.  Rather,  the  CET  seemed  to  prefer  that  these  issues  be  taken  care  of  by  someone  else, 
regarding  them  as  a  low-level  support  function  in  contrast  to  an  important  part  of  how  to 
articulate  and  express  their  work.  In  reality,  such  issues  span  the  spectrum,  from  formulating  the 
conceptual  basis  by  which  work  can  be  described  to  the  reality  of  software  ‘button-ology’. 
Further,  various  individuals  seemed  somewhat  inclined  to  address  only  the  CEE  elements  that 
were  directly  relevant  to  their  role  and/or  immediate  need.  For  example,  some  members  had 
difficulty  in  recognizing  the  relevance  of  one  tool  versus  another,  as  well  as  the  relationship 
between  certain  tools  and  parts  of  the  process.  The  lack  of  such  a  global  perspective  would  seem 
to  be  linked  either  to  an  incomplete  ‘big  picture’  understanding  of  the  CEP  (as  mentioned 
previously)  or  as  representative  of  an  intellectual  bias  in  terms  of  engineering  and  analysis 
techniques.  Irrespective  of  cause,  however,  the  CET’s  lack  of  positive  engagement  with  the  CEE 
served  to  further  a  negative  feedback  loop  in  terms  of  its  use  (i.e.,  created  a  self-fulfilling 
prophecy  of  problem  abetting  problem).  Indeed,  legitimate  concerns  were  sometimes  used  as 
excuses  when  the  team  became  frustrated  within  the  exercise,  be  it  in  terms  of  process  or 
technology.  Hence,  lack  of  interest  and  enthusiasm  further  clouded  technical  issues  (such  as 
performance)  and  the  ability  to  respond  to  them  in  terms  of  CET  satisfaction. 

Going  forward,  a  much  more  cohesive  approach  and  a  clearer  delineation  of  personnel  with 
respect  to  technology  use  will  be  required,  as  will  a  stronger  support  capability  to  assist  in 
problem  areas.  It  is  recommended  that  a  means  for  enabling  and  advocating  suitable  use  of  the 
CEE  be  made  integral  to  the  CET  structure.  This  integration  could  be  provided  either  by  a 
dedicated  CEE-focused  team  member,  or  through  multiple  team  members  whose  CEE-focused 
responsibilities  and  skill  sets  are  stringently  incorporated  into  their  role  definitions  (a  potential 
application  of  the  Team  Charter). 

5.1 .3.6  Technology,  Facilities  and  Infrastructure 

The  realities  of  distribution  and  security  in  terms  of  the  Exercise  Gamma  CEE  provided  a  number 
of  challenges  both  for  the  CET  as  end  user  and  the  Evaluation  Team  (Materiel  Advisor)  as  the 
responsible  interface  between  the  CET  and  the  environment’s  implementers  and  service  providers 
(e.g.,  various  network  and  facility  personnel).  Certainly,  a  number  of  legitimate  technical  issues 
vexed  the  CEE;  however,  a  variety  of  those  challenges  were  of  consequence  primarily  due  to  the 
prototypical/experimental  nature  of  the  exercise.  Therefore,  it  is  necessary  to  differentiate 
between  the  two  contexts,  in  order  to  address  the  actual  underlying  issue  and  not  be  sidetracked 
by  situation-specific  irritants. 

It  is  also  noteworthy  to  consider  that  the  unlike  the  CET  and  CEP,  the  CEE  is  unique  in  that  to 
some  degree,  it  was  an  integrated  and  cumulative  interface  to  a  conglomeration  of  systems  owned 
and  under  the  control  of  various  disjoint  organizations.  For  example,  the  CEP  was  a  construct 
defined  and  owned  by  the  CapDEM  TDP;  therefore,  the  Evaluation  Team  was  naturally  aware  of 
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its  status  and  evolution.  The  CEE,  however,  was  a  combination  of  tools  and  technologies  that 
existed  on,  and  communicated  over,  a  variety  of  infrastructures  (i.e.,  servers,  networks,  etc.).  Due 
to  the  large  number  of  stakeholders  involved  in  realizing  the  CEE,  various  idiosyncrasies  of,  and 
dependencies  between,  components  and  the  supporting  infrastructures  were  not  always  clear  or 
recognized.  By  virtue  of  this  reality,  the  resiliency  of  the  CEE  was  very  difficult  to  ensure. 

Within  Exercise  Gamma,  the  primary  technology  themes  included: 

•  Realistic  application.  Who  uses  specific  tools  (i.e.,  which  roles  and  with  respect  to  which 
responsibility  or  function)  and  how  the  tools  fit  with  the  process  (i.e.,  stage,  function,  etc.). 

•  Functional  specification  and  adaptation.  The  need  to  be  able  to  flexibly  adapt  the  tool 
suite  by  adding  analytical  tools  as  they  become  required  is  important.  Further,  there  needs  to 
be  clarity  on  missing  functionality  and  how  to  satisfy  unique  CF/DND  considerations  using 
standard  tools  (e.g.,  how  to  appropriately  address  PRICIE). 

•  Performance  and  usage  experience.  The  need  to  ensure  suitable,  equitable  and  resilient 
performance  for  all  tools,  including  awareness  of  how  potential  issues  at  the 
technology/policy  divide  can  significantly  impact  the  usage  experience  (e.g.,  unsatisfactory 
tool  performance  primarily  due  to  cross-network  policies). 

•  Access  and  availability.  There  is  a  need  for  readily  available  and  flexible  access  to 
specialized  facilities  (such  as  the  ACCESS  Lab)  as  well  as  the  desire  for  external  access  to 
the  CEE  (i.e.,  from  outside  the  lab)  through  the  realization  of  a  ‘virtual  facility’. 

In  terms  of  resiliency,  the  cross  infrastructure  deployment  of  the  CEE  within  Exercise  Gamma 
was  highly  problematic.  While  this  issue  was  specifically  exacerbated  by  the  prototypical  and 
experimental  nature  of  the  exercise,  ample  and  ongoing  effort  will  be  required  to  minimize  usage 
and  compatibility  problems  in  future  environments  while  still  affording  users  a  fluid,  intuitive  and 
dependable  CEE.  A  key  issue  impacting  the  availability,  performance  and  reliability  of  the  CEE 
was  the  lack  of  control  and  awareness  over  the  multiple  infrastructures  (i.e.,  networks,  facilities, 
etc.)  utilized  by  the  CET  (see  Figure  8).  The  need  to  use  multiple  networks  and  provide  multiple 
entry  points  to  the  CEE  for  purposes  of  location-independent  access  (including  the  Internet) 
proved  very  problematic,  as  each  new  access  point  inferred  a  new  network  infrastructure  with 
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which  to  be  compatible.  As  an  example,  many  of  the  difficulties  experienced  within  Exercise 
Gamma  were  due  to  unknown  technical  requirements  with  certain  software  components 
(i.e.,  implementation  side  effects),  combined  with  a  ‘trust’  issue  between  the  CEE  host  network 
(DRENet)  and  the  CET’s  primary  point  of  presence  network  (DWAN).  In  particular,  the  primary 
non-host  network  used  by  the  CET  implemented  ad  hoc,  unannounced  policy  changes  that 
prevented  a  number  of  tools  from  functioning  correctly  and/or  creating  significant  performance 
problems. 

The  above  effects  were  further  confounded  by  the  mixed  use  of  unclassified  and  classified 
facilities/networks,  their  corresponding  policies  and  procedures,  as  well  as  a  lack  of  familiarity 
and  comfort  in  terms  of  their  use.  The  key  challenge  with  the  mixed  use  of  unclassified  and 
classified  facilities/networks  was  that  certain  information  could  only  be  stored  and  processed 
within  a  classified  setting.  However,  due  to  the  limited  ability  to  readily  and  freely  work  in  a 
classified  setting,  there  was  the  need  (and  strong  benefit)  to  do  as  much  as  possible  using  the 
unclassified  CEE.  Therefore,  the  issue  of  consistency  and  linkage  (in  terms  of  analysis  and 
information)  between  the  two  security  levels  was  arduous  and  required  deliberate  effort  to 
maintain  and  utilize  both  systems  in  a  logically  cohesive  yet  physically  disjoint  manner.  Short  of 
providing  sufficient  classified  facilities  to  enable  CETs  to  do  all  their  work  in  a  classified  manner, 
there  will  continue  to  be  a  need  for  simpler  (i.e.,  automated)  and  approved  methods  to  facilitate 
exchanges  between  environments  of  differing  classifications.  While  the  issue  certainly  affected 
Exercise  Gamma,  addressing  it  was  outside  the  scope  of  the  exercise.  Indeed,  such  an  issue  offers 
significant  technical  and  policy-related  challenges,  and  is  important  to  many  efforts  within  and 
between  government  organizations.  Consequently,  the  challenges  in  this  area  will  need  to  be 
addressed  by  a  broad  range  of  R&D  initiatives. 

The  overall  result  was  a  less  than  confident  user  experience.  Ultimately,  the  desire  for  a  complete 
and  fluid  virtual  collaborative  environment  was  not  met.  Hence,  short  of  providing  cross 
infrastructure  service  level  agreements  including  a  means  for  suitable  notice  and  redress  of 
infrastructural  changes,  a  recommendation  is  that  the  CEE  should  exist  on,  be  managed  and 
supported  by,  a  single  network  infrastructure  (not  precluding  differences  due  to  required  security 
classification).  Correspondingly,  the  CET  would  benefit  from  working  solely  on  that  particular 
network  for  purposes  of  utilizing  the  CEE.  External  access  to  the  CEE  remains  desirable,  albeit 
limited  by  security  considerations  of  potential  end  points.  Moreover,  the  variety  and 
configuration  of  access  points  needs  to  be  limited  and  well-controlled  as  a  matter  of  pragmatism. 

Beyond  the  desire  for  ‘commonality’  across  security  caveats,  there  was  the  desire  for 
‘commonality’  amongst  the  various  components  of  a  given  CEE,  to  promote  the  seamless 
transition  between  them  (i.e.,  transparent  integration).  For  example,  there  was  the  request  for 
more  automated  interaction  between  document  preparation  and  architecture  tools,  so  that  there 
was  less  manual  intervention  required.  Some  progress  on  the  data  interchange  issue  had  been 
made  during  earlier  CapDEM  efforts  [13][15];  however,  issues  of  scale  and  resources  did  not 
enable  the  provision  of  this  type  of  support  within  the  classified  environment  in  time  for  the 
exercise.  Achieving  smooth  information  exchange  was  also  made  difficult  due  to  the  variety  of 
sources  and  sinks  used  as  a  result  of  security,  policy,  geographical  and  technical  issues.  For 
example,  as  highlighted  in  Section  4.2.3. 1,  a  variety  of  means  (from  printed  copy  to  swappable 
hard  drives)  were  used  in  response.  This  variety,  combined  with  a  lack  of  familiarity  in  terms  of 
relevant  information  handling  practices,  lowered  the  fluidity  and  increased  the  disjointedness  by 
which  information  was  exchanged. 
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A  number  of  other  ‘growing  pains’  and  ‘side  effects’  were  also  experienced  in  terms  of  the 
classified  ACCESS  Lab.  While  the  users  expressed  that  they  liked  the  actual  facility  (albeit  less 
aesthetically  pleasing  than  the  unclassified  lab  and  a  bit  barren  due  to  security  requirements),  the 
following  were  noted:  (1)  due  to  a  change  in  the  video  system  from  that  recommended  by  the 
Materiel  Advisor,  the  implemented  video  system’s  performance  was  not  satisfactory,  and  as  a 
result,  a  lower  quality  (visual)  experience  was  had  by  the  CET20;  (2)  physical  access  to  the  CEE 
equipment  (e.g.,  servers  and  network  equipment)  via  the  (classified)  ACCESS  Lab  was 
problematic  as  it  disturbed  meetings  and  work  sessions,  was  physically  awkward  in  terms  of  room 
logistics,  and  caused  concern  from  users  in  terms  of  interrupting  classified  activities;  hence,  better 
segregation  of  spaces  and/or  separate  entrances  need  to  be  provided;  (3)  the  desire  for  remote 
access  to  the  (classified)  CEE  through  a  ‘virtual  facility’  was  stronger  than  anticipated,  in  part  due 
to  the  CET’s  dislike  for  being  facility-dependent  (both  physically  as  well  as  in  terms  of  enclave 
configuration).  Notably,  this  issue  was  not  encountered  in  the  original  unclassified  environment; 
further,  the  use  of  different  enclave  configurations  effectively  results  in  a  modal  system,  which 
generally  suits  a  more  constrained  and  regulated  environment  -  but  at  the  expense  of  ease  of  use, 
agility,  and  ad  hoc,  creative  application. 

The  importance  of  these  kinds  of  experiential  issues  along  with  the  performance  of  the  computing 
environment  (i.e.,  hardware  and  software  infrastructure)  is  that  they  directly  influence  user 
acceptance  and  exploitation  of  end-user  applications.  As  an  experiment,  the  results  of  Exercise 
Gamma  are  best  interpreted  as  indicative  of  the  kinds  of  concerns  that  will  need  addressing  in  the 
early  days  of  an  institutionalized  CEE  so  as  not  to  skew  or  improperly  bias  its  acceptance  by  the 
user  community.  As  in  this  exercise,  when  users  are  relatively  new  to  an  approach,  its  value  and 
utility  are  still  open  to  interpretation  by  the  user  community.  Given  the  complementary  role  of 
CapDEM’s  axes,  the  experience  (and  value)  of  each  axis  and  the  effort  as  a  whole  are  impacted 
transitively.  For  example,  the  relative  success  of  distributed  teaming  issues  (in  terms  of  function 
and  work  practices)  are  influenced  by  the  suitability  and  success  of  the  communications  and 
collaboration  technologies  employed  within  the  CEE.  Conversely,  the  organization  of  team 
members  influences  what  is  required  of  the  CEE  to  enable  them  to  apply  their  part  of  the  process. 
Consequently,  as  low-level  computing  infrastructure  affects  CEE  end-user  application 
performance,  which  transitively  impacts  acceptance  and  use  of  the  CEE  and  its  success  in  helping 
the  CET  execute  the  process,  the  infrastructural  foundations  that  are  generally  outside  the  scope 
of  the  CEE  are  critically  important  to  its  success  and  that  of  the  larger  effort.  As  an  example, 
consider  Livelink,  whose  performance  was  significantly  impaired  due  to  various  infrastructural 
issues  outside  the  CEE  (Section  4.2. 3. 5).  Specific  feedback  denoted  a  sense  of  frustration  and 
hesitation  by  its  users:  “Livelink  did  not  have  performance  needed  -  always  several  seconds  per 
click;  [it]  can  not  keep  up  with  people’s  thoughts;  either  the  tool  is  bad,  installed  wrongly, 
configured  wrongly  or  infrastructure  is  problematic”.  While  tool  selection  may  have  addressed 
certain  difficulties,  the  performance  of  Livelink  within  a  single  network  indicated  the  tool  was  not 
‘bad’.  Further,  its  installation  and  configuration  were  per  specification;  however,  the  means  by 
which  tool  was  accessed  (i.e.,  infrastructure  and  access  configuration)  seemed  at  odds  with  the 
tool  (e.g.,  was  fragile  and  too  easily  ‘broken’).  Hence,  this  ‘downstream’  technical  compatibility 
issue  had  significant  impact  on  user  acceptance  (i.e.,  ‘buy-in’).  No  matter  which  tool  is  used  for 
information  management,  the  users’  infrastructure  and  broader  environment  must  not  impede  its 
function  so  the  user  community  can  properly  assess  its  worth.  Otherwise,  transitory  and  isolated 


20  Indeed  the  issue  was  significant  enough  that  lab  management  decided  to  revert  to  the  originally  proposed 
video  system  design;  unfortunately,  the  issue  was  not  remedied  within  Exercise  Gamma’s  timeline. 
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problems  with  the  technical  environment  can  inappropriately  bias  users’  perceptions  of  the 
technology  while  also  colouring  their  view  of  the  other  axes. 

Not  surprisingly,  and  as  in  previous  exercises,  there  was  a  demand  for  increased  IT  support  from 
the  CET.  While  the  specifics  of  the  requests  aligned  to  the  technical  challenges  discussed  above, 
it  is  notable  that  the  demand  for  such  support  did  not  result  from  the  specific  technical 
environment  itself  (as  there  were  different  CEE  implementations  between  exercises).  Rather,  the 
demand  tended  to  align  according  to  the  CET’s  familiarity  and  comfort  with  the  CEE,  both  as  a 
group  and  as  individual  members  (such  as  skill  set,  aptitude,  role  and  demographics).  In  reality, 
the  demand  for  support  will  always  exist  due  to  the  changing  CET  membership  and  their 
familiarity  with  the  process  and  the  specific  environment.  Within  the  exercises,  a  significant 
problem  was  that  no  actual  IT  people  were  available  in  a  dedicated  manner;  rather,  ad  hoc  IT 
personnel  were  ‘borrowed’  according  to  availability.  Consequently,  such  personnel  were  not 
necessarily  aware  of  CEE  specifics  in  sufficient  detail,  such  that  common  support  tasks  routinely 
took  longer  and  involved  more  people  than  usual.  Such  a  situation  was  obviously  less  than  ideal 
and  often  frustrated  those  involved,  thus  unfortunately  influencing  the  users’  perceptions  and 
acceptance  of  the  environment.  Going  forward,  this  issue  will  become  less  of  a  problem  as  the 
CEE  is  integrated  into  the  departmental  IT  infrastructure. 

In  the  same  way,  time  and  experience  will  highlight  and  focus  those  technology  issues  that  are 
more  endemic  to  the  particular  approach,  such  as  knowledge  and  information  management 
(e.g.,  collaborative  vs.  individual  documents,  relationship  versioning,  configuration  management, 
and  artefact-based  report  generation)  and  those  that  will  need  instance-specific  consideration 
(such  as  demographics  and  scheduling).  Demographic  influence  was  seen  across  the  various 
experiments21,  showing  up  in  tool  selection  and  usage,  as  well  as  how  the  group  worked  together 
with  respect  to  technical  issues.  For  example,  Exercise  Gamma  was  the  only  exercise  to  use  the 
document  projector  (a  near  middle-aged  military  member  who  employed  paper  documentation  to 
deal  with  security  issues).  The  demographics  (at  a  team  level)  corresponded  to  a  difference  in 
how  the  CET  would  respond  to  technical  issues.  For  Exercise  Gamma,  the  CET  did  generally  not 
try  as  a  team  to  solve  technical  problems.  Rather,  they  wanted  support  personnel  to  assist  them  in 
that  regard.  This  approach  was  in  stark  contrast  to  Exercise  Alpha  (who  engaged  the  technology) 
and  Exercise  Beta  (who  gave  up  quickly  and  did  things  manually).  These  variations  aligned  to 
how  exploratory  the  individual  teams  were,  including  tool  use.  In  Exercise  Gamma,  CET 
members  sometimes  seemed  isolated  in  their  technological  issues  (it  was  observed  that  team 
members  seemed  to  help  others  at  group  meetings  rather  than  off-line)  and  it  was  sometimes  too 
easy  to  blame  the  environment  and  use  the  classification  issue  as  a  pretext  for  not  resolving  the 
problem  at  hand.  Consequently,  this  behaviour  raised  concerns  over  CET  communication, 
particularly  in  terms  of  holistically  facilitating  collaborative  work  practices  versus  exchanging 
information  and  analysis  for  output  purposes.  Further,  as  the  CET  increased  its  focus  on  the 


21  Exercise  Alpha’s  CET  consisted  of  entirely  of  DRDC  scientists,  engineers  and  other  technical-types; 
their  general  approach  could  be  described  as  ‘experimental’.  Exercise  Beta’s  CET  was  a  blend  of  civilian 
departmental  staff  (primarily  DRDC  scientists  and  engineers)  and  various  contractors  with  a  military  lead; 
their  general  approach  could  be  described  as  ‘do  it  this  way’.  Exercise  Gamma’s  CET  was  a  blend  of 
military  and  civilian  departmental  staff  (science/engineering  and  project  management  background)  and 
contractors  with  a  military  lead(s);  their  general  approach  could  be  described  as  ‘just  get  it  done’.  The 
departmental  staff  in  Exercise  Beta  and  Exercise  Gamma  drew  increasingly  from  DRDC  CORA  (as 
opposed  to  other  DRDC  centres).  Note  that  a  number  of  the  contractors  employed  within  Exercise  Beta 
and  Exercise  Gamma  were  also  former  CF  members. 
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validation  of  its  products  (i.e.,  completing  the  system  views  properly)  near  the  end  of  the  exercise, 
the  scheduling  did  not  afford  them  have  the  opportunity  to  suitably  appreciate  the  tools  or  their 
utility.  Consequently,  the  ability  to  extract  specifics  in  terms  of  tool  use  was  inhibited  (see 
Section  5. 1.3.2).  Ironically,  the  sense  of  urgency  that  may  have  helped  the  Process  axis 
(Section  5. 1.2.1)  generally  inhibited  or  hindered  CEE  application,  such  that  the  CET  utilized  the 
‘path  of  least  resistance’  due  to  the  time  shortage  rather  than  necessarily  finding  the  best  way  to 
accomplish  the  task. 

5.2  Approach  and  Methodology 

The  following  section  provides  further  discussion  of  selected  methodological  aspects  of  Exercise 
Gamma,  specifically  the  areas  of  training  and  the  Evaluation  Team. 

5.2.1  Training 

Based  on  the  first  two  evaluation  exercises,  the  training  component  in  Exercise  Gamma  was 
formulated  with  the  intent  to  provide  hands-on,  participatory  training  across  the  various  axes. 

In  a  typical  exercise/experiment,  the  often  minimal,  abbreviated  and  sometimes  inappropriately 
customized  training  fails  to  meet  expectations.  To  some  degree,  this  was  true  of  Exercise 
Gamma.  It  is  important  to  realize,  however,  that  ‘burst’  training  cannot  be  expected  to  provide 
proficiency;  that  is,  such  training  does  not  equate  to  expertise.  Given  the  future 
institutionalization  of  the  CapDEM  approach,  a  suitable  and  sustainable  training  programme  will 
need  to  be  developed  in  conjunction  with  the  appropriate  departmental/governmental 
organizations  (e.g.,  Canadian  School  of  Public  Service  (CSPS)),  based  in  part  on  experiences  and 
lessons  learned  from  actual  projects.  Further,  many  of  the  issues  related  to  the  provisional  nature 
of  the  training  provided  in  the  evaluation  exercise  will  less  prevalent.  Key  elements  to  include 
would  be: 

•  Utilize  formal  training  staff.  Professional  trainers  (i.e.,  people  that  know  how  to  train) 
must  be  involved  (both  in  development  and  delivery  of  the  training).  Further,  if  multiple 
trainers  are  involved,  it  is  important  to  ensure  a  unified,  consistent  and  cohesive  picture 
within  and  across  the  axes  to  avoid  confusion  by  the  CET.  Within  Exercise  Gamma,  ad  hoc 
personnel  provided  the  training,  resulting  in  well-intentioned  but  generally  inadequate 
training  with  differing  (sometimes  conflicting)  trainer  viewpoints. 

•  Balance  training  relative  to  individual  needs.  Within  Exercise  Gamma,  individual 
members  of  the  CET  had  varying  familiarity  with  the  various  axes  and  their 
aspects/components.  Consequently,  the  uneven  levels  of  training  amplified  the  need  for 
follow-on  coaching.  Therefore,  the  ability  to  provide  more  flexibility  in  terms  of  individual 
training  needs  would  be  useful. 

•  Develop  a  fuller  and  more  robust  programme  (i.e.,  syllabus).  Within  Exercise  Gamma, 
the  training  programme  and  materials  were  limited  due  to  the  experiment’s  timeline  and 
Evaluation  Team  resources.  Going  forward,  the  provision  of  a  more  complete  training 
programme  with  suitable  materials  will  be  essential  to  ensuring  the  appropriate  depth  and 
breadth,  the  ability  to  adapt  to  individual  needs,  as  well  as  affording  more  flexibility  and 
realism  in  terms  of  delivery. 
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As  discussion  points,  consider  the  following  issues  in  terms  of  audience  focus  and  relevance 
within  Exercise  Gamma. 

•  Scheduling  changes  negatively  impacted  the  planned  training.  Specifically,  while  the  initial 
training  plan  was  appropriate,  the  CET  delayed  the  beginning  of  the  Elaboration  Stage  by 
three  months,  resulting  in  the  need  to  modify  the  training  plan  to  ensure  timely  delivery  (in 
accordance  with  the  PLA).  Even  so,  the  CET  received  Elaboration  Stage  training  almost  a 
month  prior  to  its  actual  start,  resulting  in  recall  difficulties.  Correspondingly,  a  noticeable 
change  in  CET  sentiment  was  detected  (i.e.,  very  enthusiastic  after  training,  less  enthusiastic 
at  the  focus  group). 

•  For  Exercise  Gamma,  the  training  component  was  modified  to  be  more  hands-on,  resulting 
in  more  engaged  participants,  increased  questions,  and  improved  training  dynamics.  Some 
training  sessions,  however,  were  still  not  particularly  participatory  (notably,  the  Process 
training). 

•  The  CET’s  preference  was  for  more  context  specific  training,  applied  relative  to  tasks  that 
specific  roles  had  to  perform,  including  what  tools  and  how  they  would  be  used  for  those 
tasks.  Some  individuals  wanted  ‘fill  in  the  blank’  training,  while  yet  others  advocated  the 
addition  of  case  studies. 

•  There  was  a  need  to  establish  and  maintain  ‘situational  awareness’  for  the  CET  in  terms  of 
the  process  at  every  training  session  (i.e.,  provide  a  global  overview  -  “where  we  are;  what 
was  done;  what’s  coming  next”).  That  is,  there  was  a  need  to  identify  and  reinforce  a 
linkage  between  what  the  CET  was  doing  and  the  ‘big  picture’  (i.e.,  capability  gap).  Indeed, 
despite  the  effort  to  do  so,  the  Inception  Stage  and  associated  training  failed  to  convey  that 
perspective;  that  is,  the  CET  was  trained  on  the  ‘big  picture’,  but  they  did  not  identify  with 
it.  Maintaining  this  linkage  was  further  challenged  by  the  use  of  an  evolving  process 
(including  late  workflow  availability).  While  such  a  developmental  issue  will  lessen  over 
later  applications  of  the  approach,  the  question  remains  how  to  suitably  communicate  such 
information  in  a  way  that  will  be  meaningful  to  a  typical  CET  (e.g.,  how  to  order,  organize 
and  present  the  material,  be  it  by  stage,  phase,  artefact,  business  practice,  etc.). 

•  The  CET  was  sometimes  too  focused  on  changing  the  tool  to  meet  their  mindset  instead  of 
thinking  of  how  to  solve  the  problem  using  the  tool  as  provided22.  This  conflict  related  to 
training  in  the  need  for  broad  understanding  to  avoid  ‘tunnel  vision’  and  the  propensity  to 
inappropriately  customize  approaches  and  representations  such  that  they  don’t  inadvertently 
become  constraints  (i.e.,  suitable  levels  of  abstraction  are  required). 

•  The  CET  needed  to  be  better  informed  in  terms  of  information  collection  not  only  for 
puiposes  of  process  integration  (as  mentioned  earlier),  but  also  for  the  identification  of 
personnel  and  the  training  required.  The  tools  to  collect  such  information  (e.g.,  enterprise 
architecture  support),  as  well  as  how  to  organize  it  (e.g.,  coaching  on  information/document 
organization)  were  the  subject  of  concern. 

•  Perception,  scheduling  and  attempts  to  leverage  training  sessions  designed  for  other 
initiatives  negatively  impacted  the  training  reality.  Specifically,  the  planned  training  for  the 
Materiel  axis  was  not  completed,  nor  was  appropriate  training  achieved.  Due  to  scheduling 
pressures  and  the  desire  “to  get  going”,  certain  training  components  were  cancelled.  Further, 


In  reality,  both  directions  need  to  be  considered. 
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the  perception  of  the  CEE  as  being  ‘just  software’  resulted  in  overconfidence  and  a  lack  of 
focus  by  some  individuals.  Further,  the  attempts  to  leverage  (i.e.,  ‘piggyback’)  training 
sessions  designed  for  other  initiatives  (such  as  DoDAF  training)  resulted  in  a  range  of 
availability,  focus  and  content  suitability  issues.  Future  training  must  guard  against  these 
kinds  of  misconceptions  as  part  of  facilitating  an  appropriate  balance  in  training  between 
axes. 

5.2.2  Evaluation  Team 

The  Evaluation  Team  was  generally  well-regarded  by  the  CET,  particularly  in  terms  of  being 
“very  responsive”  to  their  questions  and  issues.  Conversely,  the  CET  did  not  appreciate  the 
Evaluation  Team’s  style  of  interaction,  in  which  feedback  often  was  given  via  prompting  the  CET 
with  guiding  questions  as  part  of  having  the  CET  address  its  own  issues  (“don’t  answer  a  question 
with  a  question”).  While  the  approach  taken  by  the  Evaluation  Team  was  primarily  to  avoid 
undue  interference  in  the  evaluation  by  being  too  prescriptive,  the  CET  had  a  difficult  time 
appreciating  that  perspective. 

Another  source  of  consternation  was  the  nature  and  purpose  of  the  Evaluation  Team,  such  that 
there  was  confusion  over  its  role  well  into  the  execution  of  the  exercise  (i.e.,  early  on  into 
December).  In  particular,  the  term  ‘Evaluation  Team’  caused  discomfort  as  the  CET  did  not  feel 
they  knew  what  was  being  evaluated.  In  particular,  the  CET  was  concerned  that  their 
performance  was  being  assessed  (in  terms  of  correct  application  of  the  CapDEM  approach).  In 
actuality,  the  Evaluation  Team  was  assessing  the  CET’s  experience  in  applying  the  CEP  using  the 
CEE.  In  this  sense,  the  CET  was  a  ‘Subject  Team’  for  characterizing  their  experience  from  the 
CapDEM  point  of  view,  not  whether  the  CET  was  right  or  wrong  in  their  actions.  Unfortunately, 
the  CET  had  difficulty  accepting  that  perspective;  further,  they  also  found  it  difficult  to 
disassociate  between  the  same  individuals  serving  both  as  advisors  as  well  as  evaluators.  Indeed, 
at  one  point  the  CET  Lead  did  state  he  was  not  sure  why  certain  persons  were  in  the  room;  hence, 
role  clarification  was  a  crucial  issue,  illustrating  the  need  for  a  clearer  separation  of  “who  does 
what”,  both  inside  and  outside  the  CET  structure  itself.  Going  forward,  this  issue  will  diminish  as 
the  evaluator  role  will  be  transition  to  (i.e.,  be  subsumed  by)  an  advisory  and  coaching  function 
(i.e.,  a  support  team  across  all  axes  to  facilitate  training,  coaching  and  problem  solving;  to  assist 
at  the  administration/analyst  interface  (e.g.,  data  modelling)  and  assist/work  with  the  CET 
coordinator). 

Despite  those  concerns,  an  essential  function  of  the  Evaluation  Team  within  Exercise  Gamma 
was  to  provide  coaching  services  to  the  CET.  Indeed,  the  CET  identified  coaching  as  critical  to 
their  success  (“it’s  a  requirement”).  Accordingly,  three  important  aspects  of  coaching  became 
evident: 

•  Appropriate  and  suitable  roles.  There  needs  to  be  clearer  separation  of  “who  does  what” 
such  that  the  CET  found  it  confusing  when  the  same  person  would  play  several  different 
roles.  Specifically,  the  small  group  of  people  involved  in  the  exercise  often  had  to  perform 
multiple  roles  and  functions,  simply  due  to  personnel  logistics  and  resource  availability. 
This  concern  would  be  moderated  over  time  as  the  evaluation  function  is  lessened  (see 
above)  and  a  broader  base  of  SMEs  and  personnel  become  available  as  the  approach 
becomes  more  ingrained  within  the  department.  Additionally,  the  provision  of  facilitators 
during  meetings  and  other  working  sessions  was  noted  as  potentially  helpful  (although  it  was 
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unclear  to  what  degree  they  were  necessary,  how  they  would  have  been  used  and  how  they 
would  differ  from  topic- specific  SMEs). 

•  Suitably  skilled  personnel.  Suitable  training  staff  (i.e.,  people  that  know  how  to  train)  must 
be  involved  in  the  preparation  and  delivery  of  the  various  training  elements.  Similarly, 
coaching  should  be  provided  by  experienced  persons,  not  simply  using  external 
contractors/developers  to  provide  context-independent  training. 

•  Style  and  degree  of  interaction.  The  CET  preferred  a  style  of  coaching  that  was  both  pro¬ 
active  and  offered  immediate  adjudication.  They  wanted  the  coaches  to  help  frame  the 
meetings,  to  be  actively  involved  in  them,  as  well  as  to  give  near  immediate  feedback 
(i.e.,  come  back  with  answers  within  or  shortly  after  the  meeting).  While  the  original 
coaching  style  was  not  as  expected,  actions  taken  by  the  Evaluation  Team  (early  in  2007)  to 
change  the  style  and  availability  of  coaching  appeared  to  be  successful.  Specifically,  a 
variety  of  circumstances  allowed  a  limited  degree  of  ‘co-coaching’  (i.e.,  one-on-one 
coaching),  specifically  in  terms  of  the  Process  axis  (the  rationale  for  more  coaching  being 
the  lack  of  process  maturity).  Such  an  approach  was  found  to  be  efficient,  facilitating  the 
finalization  of  the  documentation  as  well  as  enabling  access  to  a  CEP  version  that  was  still 
under  construction;  it  also  improved  contact  between  the  CET  and  the  Evaluation  Team, 
helping  to  appreciate  the  other  team’s  perspectives  (including  ongoing  work  and  the  ‘big 
picture’).  Such  an  approach,  which  was  akin  to  embedding  a  per-axis  SME  within  the  CET, 
was  resource  intensive  while  still  being  perceived  as  insufficient.  Hence,  there  was  a  need 
for  more  substantial  coaching  in  each  axis  (not  just  for  Process)  as  well  as  in  the  across-axes 
domain;  that  is,  there  needed  to  be  ‘joint’  coaching  in  which  the  interaction  between  axes 
and  how  they  would  work  together  for  specific  tasks  and  specific  points  in  time  (versus  in 
general)  would  be  covered.  Such  coaching  would  help  the  CET  understand  and  achieve  a 
tighter  integration  between  the  axes,  a  key  tenet  of  the  CapDEM  approach. 
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6  Conclusion 


Despite  the  challenges  encountered  during  Exercise  Gamma,  including  its  unplanned  schedule 
extension  and  lack  of  process  iterations,  the  exercise  was  still  considered  a  success  by  both  the 
Evaluation  Team  as  well  as  the  CET.  Indeed,  the  identification  of  many  of  those  challenges  was 
the  primary  purpose  of  the  exercise  itself. 

6.1  Report  Summary 

Exercise  Gamma  provided  the  opportunity  for  a  CET  constituted  from  actual  departmental 
personnel  to  apply  the  CapDEM  approach  on  a  real  departmental  scenario.  This  enabled  the 
Evaluation  Team  to  obtain  a  significant  amount  of  data  and  insight  with  respect  to  the  CapDEM 
approach.  In  addition  to  brief  per  axis  summaries,  Table  5  provides  a  synopsis  of  Exercise 
Gamma’s  intended  goals,  scope,  outputs  and  outcomes  as  compared  to  how  they  were  realized 
within  the  actual  exercise.  It  is  important  to  note  that  the  level  of  realization  is  not  an  indication 
of  merit  or  importance  in  the  conduct  of  the  exercise;  rather  it  indicates  the  degree  to  which  a 
particular  aspect  was  addressed,  regardless  of  reason  or  extenuating  circumstance.  Further,  in 
terms  of  ‘Outputs  and  Outcomes’,  many  useful  observations  and  lessons  learned  were  obtained 
outside  the  originally  intended  measures;  consequently,  their  actual  value  is  not  represented  in  the 
table,  as  such  measures  reflected  more  on  the  reality  of  exercise  execution  and  less  on  those 
outlined  in  the  exercise’s  original  design. 

Table  5:  Exercise  Gamma  -  Realization  of  Original  Design 


Aspect 

Description 

Realization 

Goals 

Overall 

•  Test  and  evaluate  the  CET,  CEP  and  CEE  as  applied 
to  a  realistic  problem  provided  by  an  external  client. 

HIGH 

•  Ensure  the  necessary  people,  process  and  materiel 
can  address  a  ‘real  world’  problem  when  being 
executed  by  an  appropriate  real-world  client  group. 

MEDIUM/HIGH 

People 

•  Test  and  evaluate  the  final  iteration  of  the  CET. 

MEDIUM/HIGH 

•  Put  forward  best  practices  and  lessons  learned  for  a 
comprehensive  Team  Charter,  including  the  proper 
identification  of  roles  and  responsibilities,  dynamic 
teamwork  and  collaboration  practices,  and  effective 
team  communication  practices  and  mechanisms. 

MEDIUM 

Process 

•  Test  and  evaluate  version  4  of  the  CEP,  including  its 
complete  set  of  activities  and  deliverables. 

MEDIUM/HIGH 

Materiel 

•  Test  and  evaluate  the  revised  CEE. 

MEDIUM/HIGH 
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•  Investigate  and  determine  a  reasonable  list  of 
functionalities  (required,  preferred  and  optional)  for 
CEE  institutionalization. 

MEDIUM 

Scope 

Overall 

•  Seven  month  duration. 

•  ‘Real  world’  scenario. 

HIGH 

People 

•  Complete  team  size. 

•  DND  clients  from  more  than  one  organization. 

HIGH 

Process 

•  Complete  each  CEP  v4  deliverable. 

HIGH 

•  Complete  each  CEP  v4  task  and  iterate  as  mandated 
by  the  process. 

LOW/MEDIUM 

Materiel 

•  Fully  exploit  CEE  functionalities  in  performance  of 
their  tasks. 

LOW/MEDIUM 

Outputs  and  Outcomes 

People 

•  Complete  the  Team  Charter,  incorporating  roles  and 
responsibilities,  as  well  as  teamwork  and 
collaboration  principles  for  the  CET. 

•  Put  forward  recommended  best  practices  for 
initiating  a  successful  CET. 

•  Link  successful  team  dynamics  with  CEE 
(e.g.,  ACCESS  Labs  and  Livelink)  for  enhanced 
communication  practices. 

LOW/MEDIUM 

Process 

•  Ensure  the  CET  clearly  understands  CEP  processes. 

•  Ensure  the  CET  clearly  understands  CEP 
deliverables. 

•  Identify  documentation  weaknesses  (process  and 
deliverable  templates)  to  facilitate  application  during 
institutionalization  within  the  DND/CF. 

•  Identify  training  weaknesses  to  be  addressed  when 
providing  future  training  to  DND/CF  personnel. 

•  Identify  metrics  to  measure  and  facilitate  continuous 
improvement  of  the  CEP. 

•  Identify  critical  factors  for  successful  implementation 
of  the  CEP. 

MEDIUM/HIGH 
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•  Provide  a  user-validated  list  of  mandatory/preferred 
functionalities  to  fully  enable  the  CEP  and  facilitate 
collaboration  inside  the  CET. 


Materiel 


Refine  and  validate  the  existing  workflow  between 
the  various  tools. 


LOW/MEDIUM 


•  Provide  an  updated  and  expanded  database  of 
‘Frequently  Asked  Questions’  (FAQ),  ‘Tips  & 
Tricks’  and  lessons  learned. 


People.  In  terms  of  the  People  Axis,  this  exercise  reiterated  the  importance  of  CET  composition 
and  structure,  including  the  delineation  and  maintenance  of  roles  over  the  course  of  the  effort. 
Related  themes  of  team  charter  utility,  communication  and  collaboration  along  with  the 
importance  of  team  leadership  also  emerged.  Other  themes  included  the  need  for  appropriate  and 
dedicated  personnel,  expertise  integration  and  ensuring  alignment  between  the  CET  and  CEE. 

Process.  In  terms  of  the  Process  Axis,  this  exercise  highlighted  the  need  to  address  linkages 
between  the  CEP  and  external  processes  (such  as  CBP),  address  CEP  lifecycle  clarity,  alternative 
specifications,  workflow  definitions  (i.e.,  process  flow),  as  well  as  its  linkage  to  the  CEE  and 
information  management.  Process  variability,  benchmarking  and  applicability  were  also 
highlighted.  Various  documentation  issues  as  well  as  the  need  for  further  analysis  in  terms  of  the 
relationship  between  requirements,  operational  and  SoS  architectures  were  also  highlighted. 

Materiel.  In  terms  of  the  Material  Axis,  this  exercise  confirmed  the  difficulty  inherent  to 
distributed  configuration  management  and  support  across  organizations  with  different  network 
infrastructures  which  are  not  fully  interoperable.  Moreover,  various  components  of  the  CEE  had 
unanticipated  technical  and  personnel  issues.  The  key  issues  of  understanding  and  usability, 
functional  grouping  and  alignment,  information  and  knowledge,  logistics,  personnel,  and 
technology,  facilities  and  infrastructure  were  discussed.  Infrastructure  provisioning  and 
management,  secure  distributed  collaboration,  information  management,  workflow,  external 
interface  alignment  to  other  axes,  expertise  integration,  understanding  and  usability,  as  well  as  the 
expansion  of  the  technology/capability  base  were  highlighted. 

6.2  Highlighted  Recommendations 

The  presentation  and  detailed  discussion  of  the  observations,  experiences  and  lessons  learned 
within  Exercise  Gamma  have  already  been  presented  earlier  in  this  report  (see  Sections  4  and  5). 
This  section  now  highlights  a  selection  of  recommendations  abstracted  from  those  earlier 
discussions. 

6.2.1  People 

Mandate  use  of  the  team  charter.  As  part  of  maintaining  a  cohesive  and  functional  CET  with 
an  effective  understanding  and  fulfillment  of  its  roles  by  the  appropriate  individuals,  there  must 
be  a  more  defined  link  between  the  team  charter  and  the  day-to-day  governance  and  functioning 
of  the  CET.  To  provide  and  maintain  such  linkage  between  the  team  charter  and  the  day-to-day 
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governance  and  functioning  of  the  CET,  there  needs  to  be  less  dependency  on  team  volition  to  use 
the  charter  with  its  use  being  mandated  and  it  serving  as  a  practical  and  authoritative  guide,  not  an 
ancillary  ‘good  practices’  reference.  Such  functioning  includes  the  alignment  of  roles  and 
individuals  to  specific  parts  of  the  process  (CEP)  as  well  as  the  technological  environment  (CEE). 

Utilize  appropriately  designated  and  dedicated  personnel.  The  CET  must  be  composed  of 
suitably  skilled  and  interested  individuals  who  are  appropriately  available  and  in  sufficient 
number  according  to  the  requirements  of  the  CE  effort.  Focus  on  the  effort  is  important  to  ensure 
consistent  availability  with  fewer  conflicts  in  terms  of  time  and  accessibility  for  the  variety  of 
tasks  required  of  an  individual’s  role,  be  it  from  training  and  interaction  with  SMEs  to  the 
application  of  the  process  and  the  informed  use  of  the  various  support  technologies.  Hence,  the 
participating  individuals  need  to  be  dedicated  to  the  CET  and  not  divided  between  multiple  jobs. 
Aptitude  and  expertise,  combined  with  availability  and  appropriate  role  assignments,  are  key 
elements  of  a  successful  CET.  Conflict  with  other  outside  job  or  career  interests  as  well  as 
having  to  perform  too  many  roles  (i.e.,  wearing  too  many  ‘hats’  within  the  CET)  results  in  role 
confusion  and  creates  uncertainty  in  terms  of  importance  and  priorities  that  can  impact  the  rest  of 
the  team  in  the  performance  of  their  roles  and  in  their  interactions  with  fellow  CET  members. 
The  effect  of  arbitrarily  altering  specified  positions  and  role  assignments  can  have  (and  has  had) 
negative  downstream  effects,  including  intra-team  personnel  conflicts  and  a  lack  of  effective 
work  activities  due  to  misperceptions  of  competency,  importance  and  relevance. 

Integrate  expertise  within  the  CET.  As  part  of  enabling  a  self-sufficient  CET  that  does  not 
solely  depend  on  outside  knowledge  for  routine  tasks  such  as  process  application  and  tool  usage, 
the  CET  needs  to  have  resident  expertise  not  only  on  the  problem  space  being  addressed,  but  also 
in  terms  of  the  CE  construct.  For  example,  having  integrated  CET  knowledge  would  be  useful 
for  team  management,  including  a  more  ‘natural’  use  of  the  team  charter.  Knowledge  of  the  CEP 
would  help  the  members  not  only  see  the  ‘big  picture’  of  the  effort,  including  how  the  parts 
support  and  facilitate  each  other,  but  also  help  in  the  performance  of  specific  tasks.  For  the  CEE, 
a  means  for  enabling  and  advocating  suitable  use  of  the  CEE  should  be  made  integral  to  the  CET, 
either  by  a  dedicated  CEE-focused  team  member,  or  through  multiple  team  members  whose  CEE- 
focused  responsibilities  and  skills  are  stringently  incorporated  into  their  role  definitions  (a 
potential  use  of  the  Team  Charter). 

Constructively  manage  CET  roles.  As  part  of  ensuring  a  functional  effort,  the  roles  on  the  CET 
must  be  managed  in  a  concerted,  clear  and  transparent  manner.  That  is,  the  CET  must  not  be  left 
to  flounder  or  its  structure  be  changed  in  an  ad  hoc  fashion  (i.e.,  ‘on  a  whim’).  For  example,  CET 
members  need  to  fulfill  their  roles  for  the  duration  of  the  effort,  rather  than  change  (i.e.,  take  on 
or  drop)  responsibilities  without  appropriate  consideration  of  wider  implications.  Indeed,  to 
ensure  cohesive  CET  behaviour,  it  is  important  not  to  create  confusing  functional  overlap 
(i.e.,  ensure  well-delineated  responsibilities)  or  distract  members  through  interpersonal  issues 
(e.g.,  ‘turf  war’)  that  can  have  adverse  effects  on  collaboration  and  team  dynamics;  see  ‘Utilize 
appropriately  designated  and  dedicated  personnel’  above.  As  mentioned,  the  use  of  the  Team 
Charter  would  serve  as  a  mechanism  to  achieve  this  end,  as  would  embedding  knowledge  of  the 
People  Axis  issues  within  the  CET.  Further,  aside  from  the  management  of  core  CET  roles,  it  is 
important  to  clarify  the  intent  of  CET  support  roles.  For  example,  CET  concern  over  single 
individuals  serving  the  combined  roles  of  evaluator,  advisor  and  coach  role  was  a  distraction. 
Part  of  this  diversion  was  due  to  functional  overlap  combined  with  a  lack  of  clarity  with  respect  to 
the  puipose  of  particular  roles  (specifically,  the  evaluator  role)  as  well  as  being  unclear  (and 
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comfortable)  asking  for  help  from  the  same  individuals  (in  their  advisory  and  coaching 
capacities).  While  the  role  of  evaluator  will  diminish  as  it  is  transitioned  to  (i.e.,  subsumed  by)  an 
advisory  and  coaching  function  (i.e.,  a  support  team  across  all  axes  to  facilitate  training,  coaching 
and  problem  solving),  the  need  to  know  ‘who  to  go  to’  will  remain,  as  will  the  need  to  facilitate 
good  working  relationships.  Undoubtedly,  any  embedding  of  expertise  within  the  CET  (see 
above)  will  assist  in  meeting  this  goal.  Further,  ensuring  appropriate  inter-role  linkage  will  help 
facilitate  smooth  work  practices  (for  example,  assisting  at  the  administration/analyst  interface 
(e.g.,  data  modelling)  and  abetting  the  CET  coordinator).  Moreover,  the  selection  and  continued 
availability  of  a  suitably  skilled  CET  Lead  (i.e.,  Project  Manager  (PM))  is  fundamental  to 
ensuring  these  issues  can  be  addressed  in  an  appropriate  and  timely  manner. 

Increase  and  clarify  alignment  between  CEE  and  CET.  The  common  theme  of  ‘technological 
trepidation’  both  inside  and  outside  of  Exercise  Gamma  further  necessitates  an  effective  bridging 
between  the  People  and  Materiel  axes.  Attaining  this  connection  requires  stronger  and  more 
readily  identifiable  cohesion  between  the  two  axes  combined  with  a  clearer  delineation  of 
personnel  with  respect  to  the  technology  they  need  to  use.  Further,  a  stronger  support  capability 
to  assist  in  problem  areas  is  required.  Hence,  a  means  for  enabling  and  advocating  suitable  use  of 
the  CEE  is  necessary  to  facilitate  its  effective  use  by  the  CET.  Achieving  clear  alignment  to  and 
effective  usage  of  the  CEE  affects  both  the  team  as  a  whole,  as  well  as  individual  members. 
Notably,  the  performance  of  other  team  members  can  influence  interactions  between  fellow  CET 
members,  thus  impacting  team  dynamics  and  consequently,  the  efficacy  of  their  teamwork  and 
collaboration.  Consequently,  both  aptitude  and  expertise  along  with  availability  and  appropriate 
role  assignment  must  be  considered  as  part  of  achieving  this  alignment.  To  that  end,  one  way  of 
improving  the  alignment  between  the  CEE  and  CET  is  to  embed  expertise  on  the  environment 
within  the  team  structure  (as  described  above).  By  doing  so,  the  increased  self-confidence 
enables  the  CET  to  effectively  and  assuredly  use  and  explore  novel  application  of  the  tools  at 
their  disposal.  Consequently,  they  are  able  to  function  more  independently  and  not  require 
continuous  interaction  with  outside  expertise,  which  could  excessively  disrupt  their  workflow. 
Rather,  the  team  would  benefit  from  increased  self-sufficiency  and  higher  productivity. 

6.2.2  Process 

Formalize  identifiable  linkages  between  CEP  and  CBP.  Despite  the  existence  of  linkages 
between  CBP  and  the  CEP  used  within  Exercise  Gamma,  they  were  not  easily  identified  by  the 
CET  members.  As  a  result,  there  was  ongoing  apprehension  by  the  CET  as  to  how  to  proceed  in 
a  coordinated  manner  that  respected  both  processes.  Consequently,  future  CEP  versions  must 
formalize  such  linkages  and  make  them  more  easily  identifiable.  Clarifying  the  connectivity 
between  both  processes  is  important  to  ensure  both  real  and  perceived  linkage  issues  are 
addressed  and  will  be  an  important  part  of  facilitating  its  introduction  to  process  newcomers. 
More  effective  training  and  a  larger  and  broader  user  base  combined  with  increased  experience 
and  heightened  expertise  will  also  assist  with  this  issue.  Furthermore,  when  the  development  and 
utilization  of  the  process  become  under  the  purview  of  a  single  organization  (i.e.,  CFD), 
synchronization  between  the  processes  will  undoubtedly  become  more  straightforward. 

Clarify  CEP  lifecycle  and  progression.  While  key  aspects  of  the  CEP  include  its  iterative, 
incremental  and  multi-stage  design,  understanding  how  such  characteristics  interrelate  has  proven 
a  challenge  for  various  CETs,  including  Exercise  Gamma’s.  Confusion  over  which  construct  was 
which  and  how  they  related  to  each  other  was  encountered;  the  notion  of  performing  the  entire 
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series  of  stages  within  each  iteration  was  an  example  of  such  a  misunderstanding.  Going  forward, 
clarity  of  the  CEP  lifecycle,  including  construct  relationships  and  progression  between  them  must 
be  more  clearly  communicated;  for  example,  the  difference  between  iteration  vs.  stage  and  their 
relative  progression.  In  the  broadest  sense,  the  CET  needs  a  clear  understanding  as  how  to 
execute,  manage  and  understand  stages  in  an  iterative  and  incremental  manner.  More  effective 
training,  enhanced  documentation,  increased  experience  and  heightened  expertise  will  further 
reduce  the  impact  of  this  issue. 

Provide  alternative  CEP  specifications.  As  with  many  processes,  the  specification  of  the 
process  can  significantly  impact  how  easily  it  is  understood  and  followed.  With  the  CEP,  there 
was  a  broad  consensus  that  the  provision  of  complementary  specifications,  specifically  the  use  of 
deliverable-centric  and  task-centric  methods,  would  be  important  to  fostering  greater 
understanding  and  acceptance  of  the  process  within  the  department.  Notably,  however,  one  of  the 
challenges  of  such  a  dual-pronged  approach  is  the  need  to  maintain  coordination  and  consistency 
between  them. 

Ensure  workflow  independence.  One  of  the  challenges  in  applying  a  novel  process  is  to 
achieve  understanding  and  acceptance  (i.e.,  buy-in)  by  its  practitioners  and  relative  to  its 
supporting  technologies.  Doing  so  while  avoiding  instance-specific  explication  and  maintaining 
sufficient  independence  to  enable  broad  and  diverse  application  is  even  more  challenging. 
Workflows  should  be  specified  suitably  orthogonal  to  (i.e.,  independently  of)  the  specific  problem 
and  technical  environment;  however,  for  ease  of  use,  they  must  be  linked  in  an  illustrative 
(normative)  manner  to  the  CEE  to  better  enable  CET  understanding. 

Clarify  CEP  and  CEE  linkage  relative  to  information  management  practices.  The  linkage 
and  mutual  influence  amongst  information  modelling,  tool  input  and  output,  process  input  and 
output  (i.e.,  deliverables)  and  their  specification  relative  to  each  other  needs  considerable 
forethought  and  appropriate  structuring  in  order  to  facilitate  productive  process  application 
through  the  effective  use  information  management  and  the  various  CEE  tools.  To  date,  this  has 
been  a  problem  as  neither  tool  or  information  management  (structuring)  knowledge  was  sufficient 
within  the  CET.  A  more  well  thought-out  approach  and  the  formalization  of  these  topics  as  an 
essential  part  of  the  training  programme  would  prove  useful  going  forward.  As  mentioned  under 
the  People  discussion  above,  the  potential  embedding  of  such  expertise  within  the  CET  would  be 
advantageous.  Notably,  well-founded  information  structuring  and  management  principles  need  to 
be  applied  and  serve  as  the  basis  for  well  principled  exploration  of  (and  deviation  from)  both  the 
CEE  and  the  CEP  once  comfort  and  understanding  of  the  fundamentals  are  established. 

Investigate  deliverable  benchmarking  and  applicability  to  decision  making.  Further 
investigation  as  to  the  merit  of  creating  benchmarks  for  deliverable  completion  and  how  to 
provide  guidance  in  support  of  such  benchmarking  remain  open  questions.  While  illustrating  the 
potential  of  architectures  and  capability  engineering  within  the  force  development  process, 
Exercise  Gamma’s  CET  was  not  able  to  make  solid  conclusions  about  their  applicability  within 
the  decision  process.  Given  the  novelty  across  the  breadth  of  the  exercise  (e.g.,  new  CET 
membership,  revised  process,  and  revised/realigned  CEE)  this  is  a  realistic  outcome  that  serves  as 
an  appropriate  basis  for  additional  departmental  evaluation. 

Follow  the  process  and  accept  its  variability.  Within  the  various  applications  of  the  process, 
there  have  been  and  will  always  be  natural  variation  in  the  conduct  of  the  CEP,  in  part  due  to  its 
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descriptive  rather  than  prescriptive  specification.  The  amount  of  variance  in  Exercise  Gamma, 
however,  was  regarded  as  atypical  of  that  going  forward,  assuming  increased  process  maturity 
and  practitioner  experience.  To  that  end,  it  will  be  important  to  properly  control  process 
evolution  during  its  application  (i.e.,  avoid  utilizing  multiple  versions  of  the  process  within  a 
single  instance).  Furthermore,  it  will  be  important  to  limit  premature  workflow  customization,  so 
as  to  prevent  unnecessary  divergence  from  recommended  practices.  That  is,  it  will  be  important 
to  ensure  that  practitioners  know  and  utilize  ‘the  basics’  first. 

6.2.3  Materiel 

Ensure  reliable  infrastructure  provisioning  and  management.  A  significant  issue 
experienced  within  Exercise  Gamma  was  the  use  of  multiple  and  independently  managed 
networking  and  computing  infrastructures  (specifically,  the  DRENet,  the  DWAN  and  the 
Internet).  While  this  situation  was  an  artefact  of  TDP  reality,  it  was  an  unremitting  source  of  user 
difficulties  and  technical  challenges,  which  grew  each  time  the  exercises’  participants  became 
broader.  As  detailed  previously,  typical  difficulties  included  unplanned  and/or  unannounced 
network  policy  changes  that  adversely  affected  the  remote  use  of  tools  outside  of  the  host 
network.  Consequently,  the  ability  to  work  (i.e.,  utilize  the  CEE)  on  the  host  network  would 
eliminate  the  myriad  of  cross-network  issues.  While  external  access  to  the  CEE  remains 
desirable,  the  variety  and  configuration  of  access  points  needs  to  be  limited  and  well-controlled  as 
a  matter  of  pragmatism  (such  as  the  ability  to  technically  support  and  manage  only  a  certain 
number  of  configurations).  Further,  security  and  classification  issues  will  need  to  be  more  deeply 
considered  relative  to  the  provision  of  potential  end  points  (see  below).  Moreover,  short  of 
providing  cross-infrastructure  service  level  agreements  which  include  a  means  for  suitable  notice 
and  redress  of  infrastructural  changes,  the  CEE  should  exist  on,  be  managed  and  supported  by  a 
single  network  infrastructure  (not  precluding  differences  due  to  security  classification). 

Investigate  requirements  and  alternatives  for  secure  distributed  collaboration  using  non- 
homogenous  environments.  Further  study  is  required  as  to  the  means  to  enable  and  support 
different  internal  and  external  configurations  of  the  tool  environment(s)  relative  to  security  and 
classification  issues.  One  example  is  whether  and  in  what  manner  external  access  to  a  classified 
CEE  and/or  the  classified  ACCESS  Lab  would  be  viable.  The  value  of  said  functionality  (which 
could  be  described  as  a  secure  collaborative  VPN  environment)  is  anticipated  to  be  markedly 
increased  should  CEE  use  become  commonplace  given  the  institutionalization  of  the  CapDEM 
approach  (based  on  the  increased  frequency,  distribution  and  significance  of  such  requests).  A 
further  issue  to  be  addressed  is  whether  or  not  and  to  what  degree  multiple  external  configurations 
could  be  made  available  at  the  same  time,  independent  of  the  local  configuration  of  the  host  CEE. 
Based  on  the  initial  design,  it  is  anticipated  that  further  study  and  re-engineering  would  be 
required  to  address  this  additional  requirement. 

Increase  focus,  expertise  and  alignment  in  terms  of  information  management  and  associated 
technological  capability.  Despite  a  significant  emphasis  on  information  availability,  creation, 
storage  and  access  as  part  of  applying  the  CEP,  there  were  considerable  challenges  in  providing  a 
clear,  flexible,  extensible,  scalable,  comprehensive  and  understandable  means  to  use,  link  and 
describe  information  that  was  amenable  to  CET  usage,  CEP  application  and  CEE  representation. 
Underlying  this  situation  was  a  lack  of  suitable  CET  expertise,  SME  availability  and  a  variety  of 
technical  and  operational  issues.  The  team’s  conceptual  understanding  and  approach  to 
information  management,  along  with  the  pragmatics  of  its  implementation,  were  problematic  and 
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did  not  align  with  the  need  for  the  variable  levels  of  granularity  and  composability  required  to 
meet  the  criteria  identified  above.  Indeed,  their  achievement  requires  a  systematic  level  of 
integration  and  interoperability  between  the  axes  in  order  to  clearly  identify  role  and  technology 
alignment  to  (i.e.,  responsibility  for)  a  given  artefact;  the  identification  of  relevant  process 
elements  (be  it  deliverables,  workflows  or  portions  thereof)  and  how  they  are  linked  to 
(i.e.,  modelled,  represented  and  processed  within)  the  CEE  also  require  a  comprehensive  and 
scalable  approach.  Such  malleability  requires  a  shift  away  from  the  ‘shared  file/network  drive’ 
paradigm  towards  the  use  of  composable  information  architectures.  As  part  of  such  a  migration 
towards  information  management  and  eventually  towards  the  goal  of  knowledge  management, 
increased  expertise  from  a  variety  of  information  and  technology  fields  will  be  required 
(potentially  including  DRDC  scientific,  knowledge  management  and  IT  communities).  Notably, 
many  of  these  resources  were  not  available  during  the  exercise;  however,  as  the  CapDEM 
approach  becomes  increasingly  part  of  departmental  practices,  achieving  such  linkages  will 
presumably  be  less  difficult. 

Facilitate  workflow  independence.  As  with  the  application  of  a  novel  process,  one  of  the 
challenges  in  using  a  collection  of  unfamiliar  technologies  is  to  achieve  understanding  and 
acceptance  (i.e.,  buy-in)  by  users.  Achieving  this  goal  while  avoiding  instance-specific 
explication  and  maintaining  sufficient  abstraction  to  enable  broad  and  diverse  application  is  even 
more  challenging.  As  part  of  moving  towards  composability  and  conceptual  interoperability  [16], 
workflows  need  to  be  specified  suitably  orthogonal  to  (i.e.,  independently  of)  the  specific 
problem  and  technical  environment.  While  the  workflows  must  be  linked  in  an  illustrative 
(normative)  manner  to  the  CEE,  as  part  of  facilitating  learning  and  providing  a  baseline  for  initial 
use,  such  linkages  must  not  be  done  so  as  to  create  unnecessarily  restrictive  limits  or 
dependencies  as  each  of  the  axes  mature  and  change  over  time.  Facilitating  workflow 
independence  from  a  CEE  perspective  is  necessary  to  complement  the  process-centric  workflow 
independence  (mentioned  above).  This  decoupling  of  logical  (process)  vs.  physical 
(technological)  workflows  creates  a  more  granular  (and  therefore  flexible  and  more  adaptable) 
way  to  repuipose  and  realign  the  ‘what’  from  the  ‘how’.  As  with  the  notion  of  alternative  process 
specifications  (detailed  above),  such  an  approach  can  assist  in  the  delineation  of  responsibilities, 
therefore  informing  CET  membership  relative  to  different  process  and  technology  skills. 

Increase  support  for  interfaces  to  external  processes.  In  support  of  the  process  goal  of  clearer 
linkage  to  external  processes  (e.g.,  CBP,  upper  management,  chain  of  command,  etc.),  there  is  a 
need  to  investigate  the  technological  requirements  and  implications  of  linking  CEE  systems  to 
those  used  by  external  processes.  That  is,  this  issue  speaks  to  the  need  for  technological  support 
along  the  CEP’s  interface  to  other  departmental  processes.  Within  Exercise  Gamma,  the 
collection  of  information  in  terms  of  input  from  organizational  levels  above  the  CET  was 
difficult.  This  difficulty  led  to  the  request  for  tools  to  facilitate  the  crossing  of  organizational 
boundaries;  for  example,  the  use  of  an  enterprise  architecture  tool  to  assist  in  obtaining  and 
organizing  system  attributes.  In  working  towards  this  capability,  clearer  identification  of  relevant 
organizations,  processes  and  systems  needs  to  be  provided,  while  issues  of  representation, 
compatibility,  security  and  access  (e.g.,  permissions)  also  need  to  be  considered  (again,  related  to 
the  notion  of  conceptual  interoperability  [16]).  While  such  functionality  was  not  within  the  scope 
of  this  effort,  it  obviously  needs  future  consideration  as  the  CapDEM  approach  becomes  part  of 
the  broader  capability  management  domain.  Notably,  addressing  this  need  will  also  be  easier 
given  the  adoption  of  the  above  information  management  recommendations. 
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Clarify  CEE  support  for  information  management  practices.  As  discussed  previously,  the 
confluence  of  information  modelling,  tool  input  and  output,  process  input  and  output  and  their  use 
relative  to  each  other  needs  considerable  forethought  in  order  to  facilitate  productive  process 
application.  In  terms  of  the  CEE,  appropriate  structuring  and  use  of  information  management 
principles  applied  at  the  interface  between  the  axes  require  the  CEE  to  provide  a  composable 
interface  (as  discussed  above  in  terms  of  workflow  independence).  To  date,  neither  tool  nor 
information  management  (structuring)  knowledge  was  sufficient  and/or  available,  resulting  in 
very  superficial  interaction  with  the  technological  environment  and  its  information  handling 
abilities.  Well-founded  information  structuring  and  management  principles  need  to  be  applied 
and  serve  as  the  basis  for  well  principled  application  of  the  CEE.  Again,  the  availability  of  such 
expertise  within  the  CET  would  be  advantageous. 

Increase  and  clarify  alignment  between  CEE  and  other  axes.  Circumventing  missteps  in  the 
application  of  the  CEE  is  a  key  element  in  the  CET  being  able  to  function  more  independently 
and  avoid  excessive  workflow  disruptions  due  to  interaction  with  outside  expertise.  Ensuring 
appropriate  technical  propensity  along  with  a  balance  of  technology  cohesion  and  delineation 
relative  to  the  other  axes  would  be  part  of  achieving  clear  alignment  to  (and  effective  usage  of) 
the  CEE  both  by  the  team  as  a  whole,  as  well  as  by  individual  members.  Further,  the 
performance  of  individual  team  members  can  influence  member  interaction,  therefore  impacting 
team  dynamics  and  consequently,  the  efficacy  of  their  teamwork  and  collaboration. 
Consequently,  the  resulting  impact  on  both  individual  and  team-wide  self-sufficiency  and 
productivity  can  significantly  impede  or  advance  the  broader  engineering  effort.  Indeed,  such 
clarity  would  reduce  the  potential  of  inappropriate  technology  application,  along  with  helping  to 
focus  the  selection  of  useful  and  forward-looking  technologies. 

Integrate  CEE  expertise  within  the  CET.  The  availability  of  CEE  expertise  within  the  CET  is 
useful  not  only  to  assist  in  tool  application  and  information  management  issues  (as  discussed 
above),  but  also  to  facilitate  advocacy  of  suitable  usage,  awareness  of  potential  application  and/or 
which  pitfalls  to  avoid  (i.e.,  provide  mentorship  and  coaching).  Achievable  through  a  number  of 
possible  CET  configurations  (see  People  section  above),  it  is  notable  that  complementary  resident 
expertise  in  terms  of  the  CE  construct  would  aid  in  creating  a  holistic  understanding  and  reduce 
the  potential  for  technological  silos  (such  as  the  knowledge  of  a  particular  software  but  not  aware 
of  its  implications  within  the  broader  technological  environment  or  the  engineering  effort  itself). 

Promote  increased  understanding  and  usability.  The  themes  of  understanding  and  usability 
underlie  many  of  the  issues  that  affected  the  CEE  during  Exercise  Gamma.  In  particular,  a  lack 
of  understanding  in  terms  of  how  to  use  particular  tools  as  well  as  how  to  use  them  in  conjunction 
with  each  other  exacerbated  the  level  of  difficulty  experienced.  By  addressing  the  above  issues  of 
expertise  integration  and  improved  clarity/alignment,  individual  members  can  be  more  focused  on 
relevant  tools  and  process  concerns  (i.e.,  their  efforts  will  be  less  ‘scattered’).  Additionally,  such 
clarity  will  enable  more  focused  provision  of  training,  mentoring  and  coaching,  along  with  a 
stronger  ability  to  target  problematic  areas  in  terms  of  technical  support  and  the  evolution  of  the 
CEE. 

Expand  technology  and  associated  capability  base.  In  line  with  the  evolution  of  the  process 
and  the  user  community  that  will  be  conducting  the  Capability  Engineering  effort,  there  is  a  need 
to  explore  alternative  and  developing  technologies  (including  disruptive  technologies)  as  part  of 
providing  an  innovative  and  creative  engineering  environment  that  will  continue  to  support  the 
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changing  needs  of  the  department,  participating  staff  (e.g.,  scientists,  engineers,  SMEs,  uniformed 
members)  and  the  advancing  technological  landscape.  Such  a  ‘technology  watch’  is  necessary  as 
part  of  addressing  the  breadth  and  depth  of  tool  functionality  that  corresponds  to  an  evolving  user 
base,  their  expertise/advice  and  with  the  need  grow  beyond  the  typical  toolset. 

6.2.4  Training 

Provide  a  suitable  training  programme.  To  enable  a  sufficiently  capable  and  serviceable 
curriculum  for  Capability  Engineering  and  the  CapDEM  approach,  a  complete  self-contained 
training  programme  is  required.  Such  a  training  programme  will  require  useful  instructional 
materials  and  suitable  instructors,  offering  both  appropriate  depth  and  breadth,  the  ability  to  adapt 
to  individual  needs,  as  well  as  affording  more  accessibility,  flexibility  and  realism  in  terms  of 
delivery.  Within  the  evaluation  exercises,  the  ability  to  provide  suitable  training  was  already 
constrained  and  personnel  availability  already  made  scalability  an  issue.  Consequently,  the  use  of 
governmental  training  mechanisms,  such  as  the  Canada  School  of  Public  Service  (CSPS),  is 
suggested  to  address  the  above  concerns,  most  notably  that  of  scalability  and  resource  constraints. 
The  independence  and  separation  of  the  training  personnel  from  the  CET  and  its  support  staff 
would  help  address  the  issue  of  functional  overlap  and  the  perception  of  evaluation  in  cases  where 
coaching  and  other  ‘on  the  job'  facilitation  is  required.  Similarly,  the  development  of  an 
electronic  online  tutorial  system  is  recommended  as  a  complement  to  initial  training  efforts,  both 
as  a  compendium  and  as  the  basis  for  follow-on  reference  and  individual  self-help  retraining. 

6.2.5  Other 

Ensure  clear,  accessible  and  reliable  logistics  support.  Within  Exercise  Gamma,  logistical 
issues  detracted  from  CET  focus  and  CEE  application.  To  mitigate  these  issues  going  forward, 
clearly  identified  and  reliable  logistic  support  are  required,  including  the  need  to  ensure 
environmental  and  infrastructural  readiness  in  a  more  reliable  and  timely  manner.  The  concerns 
regarding  facilities  (e.g.,  lack  of  classified  work  places,  network  availability,  etc.)  speak  to  the 
larger  issue  of  exercise  logistics  and  the  logistics  of  applying  the  CapDEM  approach.  While 
these  two  issues  intersect,  it  is  important  to  realize  they  were  not  the  same  and  will  naturally 
differentiate  going  forward.  That  is,  it  is  anticipated  that  future  applications  of  Capability 
Engineering  as  an  institutionalized  (i.e.,  operational)  departmental  process  will  facilitate  easier 
logistical  support. 

6.3  Concluding  Observations  and  The  Way  Ahead 

Applying  a  system  engineering  approach  at  a  capability  level  is  new  to  the  Department  of 
National  Defence  and  its  constituent  organizations.  Thus  it  will  continue  to  require  further 
improvement  through  experimentation  and  application.  In  addition  to  becoming  knowledgeable 
in  the  practice  of  CE,  it  is  anticipated  that  the  construct  will  require  continuous  improvement  as  it 
evolves  into  its  niche  within  the  force  development  community.  Challenges  will  include 
institutional  resistance  to  change  (e.g.,  legacy  of  environmentally-aligned  stovepipes,  ‘not- 
invented-here’  syndrome),  and  the  availability  of  knowledgeable  personnel  that  can  be  fully 
dedicated  to  applying  the  approach. 
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As  expected  for  any  new  approach,  concerns  arise  over  its  complexity,  duration  and  which  of  its 
elements  are  truly  essential  to  supporting  the  needs  of  the  relevant  decision  makers.  Efficient  and 
effective  team  generation  with  effective  training  and  tools  continues  to  be  recognized  as  a 
required  area  and  warranting  further  improvement.  Furthermore,  the  level  of  detail  and  volume 
of  analytical  products  required  to  satisfy  capability  level  decisions  remains  an  ongoing  question 
that  may  not  be  completely  answered  until  a  use  case  is  transitioned  into  implementation  via  the 
capability  production  domain. 

This  exercise  was  the  first  attempt  to  implement  the  Capability  Engineering  process  across  the 
broader  enterprise.  While  the  participants  were  generally  satisfied  with  their  work,  they  did  not 
reach  a  sufficient  level  of  knowledge  enabling  them  to  properly  evaluate  the  usefulness  and 
adequacy  of  their  work,  therefore  limiting  their  ability  to  evaluate  Capability  Engineering  and  the 
CapDEM  approach  as  a  larger  whole.  In  general,  the  exercise  can  be  considered  a  success,  such 
that  most  of  these  elements  were  addressed  while  offering  important  insight  into  the  day-to-day 
challenges  affecting  its  institutionalization.  Further,  they  will  serve  as  important  input  to 
organizations  applying  the  approach  in  the  future.  It  is  also  important  to  note  that  the  various 
efforts  to  advance  the  CapDEM  approach  occurred  at  the  same  time  when  the  capability  planning, 
management  and  production  domains  were  being  defined  for  the  first  time.  As  with  other 
departmental  concepts  like  capability  roadmaps,  the  continuous,  iterative  and  incremental 
improvement  of  the  capability  engineering  construct  will  also  be  required,  thus  ensuring  the  need 
to  couple  its  evolution  with  an  ongoing  evaluation  strategy. 

Based  on  the  evaluation  efforts  to  date,  critical  success  factors  for  the  application  of  Capability 
Engineering  include  the  following:  commitment  by  DND/CF  executives  to  realize  all  portions  of 
the  Capability  Engineering  construct;  creation  of  specially  trained  ‘CE  officers’  to  effectively 
enable  institutionalization  and  continuous  improvement;  direct  involvement  of  the  operational 
community  into  solution  development  in  order  to  provide  operationally  acceptable  options; 
commitment  and  participation  of  relevant  capability  implementers  in  order  to  assess  option 
feasibility;  and  the  availability  of  appropriate  support  and  resources  for  the  people  and  the 
technical  environments  employed  in  making  Capability  Engineering  a  reality. 

Lastly,  while  this  exercise  was  done  within  the  Department  of  National  Defence,  it  is  quite 
reasonable  to  believe  that  the  findings  and  conclusions  will  be  applicable  to  any  large  enterprise 
willing  to  shift  toward  Capability  Engineering  and  the  CapDEM  approach. 
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Annex  A  First  Focus  Group 


A.1  Questionnaire 


Focus  Group  Discussion  Guide 
Exercise  Gamma:  Solving  a  Real  Problem 

Exercise  Gamma  is  intended  to  be  a  complete  “3rd  party”  functional  test  and  evaluation  of 
CapDEM’s  CEP,  CET  and  CEE  axes.  Based  on  a  realistic  problem  definition,  Exercise  Gamma 
shifts  emphasis  towards  external  groups  being  able  to  address  their  problem  using  the  CapDEM 
approach.  The  CET  will  be  composed  of  individuals  from  relevant  client  groups.  The  primary: 
goal  of  this  iteration  is  to  test  and  evaluate  the  CEP  via  an  instance  based  on  Force  Planning 
Scenario  2. 

To  help  understand  how  Exercise  Gamma  is  progressing  after  completing  the  Inception  Stage, 
you  are  being  asked  to  participate  in  this  focus  group  session.  This  focus  group  will  be  asked 
approximately  20  questions  (as  well  as  additional  probing  questions)  to  help  the  Evaluation 
Team  assess  the  effectiveness  of  the  three  CapDEM  axes,  training  and  Evaluation  Team. 

At  this  point,  I  would  like  to  reiterate  the  background  of  the  Inception  Stage,  including  the 
objectives,  essential  activities,  deliverables  and  milestones. 
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The  CE  Initiative  -  Inception  Stage 


Objectives 

The  main  goal  of  the  Inception  Stage  is  to  establish  the  scope  and  boundaries  of  the 
capability  need,  establish  the  engineering  team,  obtain  approval  on  the  CE  initiative 
mandate,  team  resourcing,  plans  and  budget.  The  objectives  include: 

•  Confirm  CE  initiative  mandate  and  establish  core  engineering  team  members; 

•  Establish  CE  initiative  management  plans  and  budget,  and  finalize 
engineering  team  resourcing; 

•  Establish  CE  initiative  engineering  and  management  environments; 

•  Gather  information  on  existing  operational  and  SoS  architectures  delivering 
the  capability;  and 

•  Understand  the  strategic  context,  the  high-level  operational  concepts  as  well 
as  needs  and  constraints. 

Essential  Activities 

The  major  activities  to  be  realized  during  the  Inception  Stage  are: 

•  Preliminary  analysis  of  the  capability  and  stakeholders  needs  to  establish  the 
CE  initiative  mandate; 

•  Selection  of  core  CET  members; 

•  Confirmation  of  CE  initiative  mandate; 

•  Tailoring  of  the  standard  CEP  activities  to  the  specific  nature  of  the  CEP 
mandate,  the  capability  to  be  addressed,  and  the  stakeholders  involved; 

•  Finalization  of  CET  resources  including  Operational  and  PRICIE 
representatives; 

•  Establishment  of  CE  initiative  engineering  and  management  environments, 
including  the  requesting  and  approval  of  required  resources,  infrastructures 
and  tools; 

•  Establishment  of  the  CE  initiative  execution  schedule,  including  the  detailed 
scheduling  of  the  Inception  Stage  and  subsequent  iterations; 

•  Preparation,  validation,  approval  and  publishing  of  CE  initiative  mandate, 
management  plans  and  budget; 

•  Amendment  and  acceptance  of  Team  Charter; 

•  Preparation  and  holding  of  the  kick-off  meeting; 

•  Collection  of  relevant  information; 

•  Analysis  of  strategic  factors,  and  identification  of  high-level  needs  and 
constraints  based  on  deductions; 
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•  Definition  and  prioritization  of  key  decision  criteria  (cost,  schedule,  risk, 
performance); 

•  Drafting  of  the  operational  concept  graphic; 

•  Ongoing  enforcement  of  CE  initiative  assessment  and  control  activities; 

•  Configuration  and  verification  of  the  CEE; 

•  Preparation  and  delivery  of  initial  training  sessions  on  all  axes. 

Deliverables 

The  major  deliverables  to  be  available  at  the  end  of  the  Inception  Stage  are: 

•  Engineering  Management  Plan  -  Release  1  (external  release)  including  CE 
initiative  mandate,  overall  plans  and  detailed  plan  of  the  Inception  Stage; 

•  Strategic  Context  Analysis  -  Release  1  (external  release); 

•  Operational  Architecture  -  Draft  (operational  concepts). 

Milestone 

At  the  end  of  the  Inception  Stage,  the  initial  CE  initiative  management  plans  and  budget 
shall  be  approved  and  there  should  be  a  common  agreement  and  a  shared  understanding 
of  strategic  factors,  high-level  needs  as  well  as  the  constraints  and  key  strategic  decision 
criteria. 

At  this  point,  if  the  objectives  are  met,  then  the  Comprehension  Stage  can  commence. 
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Focus  Group  Questions 
Inception  Stage 


General 

1.  Overall,  how  would  you  assess  your  participation  within  the  Inception  Stage? 

2.  Overall,  how  would  you  assess  your  achievement  of  the  objectives  within  the  Inception 
Stage? 

People 

3.  Please  identify  any  issues  that  you  may  have  encountered  with  the  specified  roles  and 
responsibilities  of  the  CET. 

4.  How  would  you  describe  the  level  of  Team  Leadership  (Team  Leader  and  Sub-Team 
Leaders)  within  the  CET? 

5.  Please  describe  how  you  incorporated  the  principles  of  teamwork  and  collaboration 
within  the  CET. 

a.  Describe  how  the  team  worked  together  to  produce  the  deliverables. 

b.  Did  you  employ  teambuilding  exercises  with  the  CET  throughout  the  past  several 
weeks? 

6.  Please  describe  how  the  Team  Charter  has  been  employed  in  your  team? 

a.  What  elements  of  the  Team  Charter  have  been  useful? 

b.  What  elements  of  the  Team  Charter  could  be  improved? 

7.  Please  describe  how  the  CET  communicated  to  ensure  an  integrated  collaborative 
approach. 

Process 

8.  Please  identify  any  process  strengths  and/or  weaknesses  regarding  the  Inception  Stage 
that  would  facilitate  its  application? 

a.  What  would  you  suggest  to  ease  this  stage?  Examples  of  improvement  are: 
enhance  description  of  existing  activities,  remove  or  add  activities,  templates, 
etc. 

b.  Were  the  activities  pertinent  to  the  stage? 

c.  Were  the  proposed  steps  coherent? 

d.  Was  the  ordering  of  the  steps/activities  coherent? 

e.  What  activities  were  done  effectively?  Why? 

f.  What  activities  were  missing? 

g.  Were  the  activities  well-described  and  understandable? 

9.  As  a  team,  or  as  an  individual,  how  would  you  assess  your  understanding  of  the  CEP 
deliverables? 

10.  Please  identify  any  documentation  strengths  and/or  weaknesses  (including  deliverable 
templates,  online  documentation,  etc.)  that  would  facilitate  application  in  subsequent 
iterations. 
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Materiel 


1 1 .  Please  describe  your  experience  with  using  the  current  CEE  in  helping  you  go  through 
the  activities  of  this  iteration. 

12.  Please  describe  the  tools  and  how  they  were  used  to  produce  the  deliverables. 

13.  Please  describe  your  experiences  with  respect  to  the  use  of  the  ACCESS  Labs  (both  at 
CORA,  Shirley’s  Bay). 

14.  Please  describe  the  CEE  functionalities  that  helped  facilitate  team  collaboration. 

a.  What  would  likely  help  in  the  future? 

15.  Please  describe  the  CEE  functionalities  that  helped  facilitate  communication  within  the 
CET. 

a.  What  would  likely  help  in  the  future? 

Training 

16.  At  this  point  in  time,  how  would  you  describe  the  level  of  training  for: 

a.  Human  dynamics 

b.  CEP  Inception  and  Comprehension  Stages 

c.  Tools 

d.  Related  topics  (including  Architecture) 

Evaluation  Team 

17.  Please  describe  the  level  of  support  received  from  the  Evaluation  Team. 

a.  What  went  well? 

b.  What  improvements,  if  any,  would  you  recommend? 
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Annex  B  Second  Focus  Group 


B.1  Questionnaire 


Focus  Group  Discussion  Guide 
Exercise  Gamma:  Solving  a  Real  Problem 

Exercise  Gamma  is  intended  to  be  a  complete  “3rd  party”  functional  test  and  evaluation  of 
CapDEM’s  CEP,  CET  and  CEE  axes.  Based  on  a  realistic  problem  definition,  Exercise  Gamma 
shifts  emphasis  towards  external  groups  being  able  to  address  their  problem  using  the  CapDEM 
approach.  The  CET  will  be  composed  of  individuals  from  relevant  client  groups.  The  primary 
goal  of  this  iteration  is  to  test  and  evaluate  the  CEP  via  an  instance  based  on  Force  Planning 
Scenario  2. 

To  help  understand  how  Exercise  Gamma  is  progressing  during  the  Comprehension  Stage,  you 
are  being  asked  to  participate  in  this  focus  group  session.  This  focus  group  will  be  asked  a 
variety  of  questions  (along  with  probing  questions)  to  help  the  Evaluation  Team  assess  the 
effectiveness  of  the  three  CapDEM  axes  and  Evaluation  Team. 

As  a  reminder,  the  context  for  this  discussion  is  the  work  accomplished  since  the  Inception  Gate 
Review  and  Focus  Group  until  now. 
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Focus  Group  Questions 
Mid  Comprehension  Stage 


General 

1 .  Overall,  how  would  you  assess  your  participation  within  the  Comprehension  Stage  to 
date? 

Process 

2.  Given  the  CEP  workflow  for  the  Comprehension  Stage  (see  slide),  please  answer  the 
following. 

a.  What  tasks  have  you  completed? 

b.  How  well  have  you  completed  them? 

c.  How  closely  did  you  follow  the  workflow  as  presented? 

d.  Did  you  augment  the  workflow?  If  so,  how? 

e.  Was  the  ordering  of  the  tasks  coherent? 

f.  Were  the  tasks  well-described  and  understandable? 

3.  Please  identify  any  documentation  strengths  (including  deliverable  templates,  online 
documentation,  etc.)  that  would  facilitate  application  in  subsequent  iterations. 

a.  What  was  your  experience  with  the  online  CEP  documentation  (i.e.,  the  web 
page)? 

4.  Please  identify  any  documentation  weaknesses  (including  deliverable  templates,  online 
documentation,  etc.)  that  would  facilitate  application  in  subsequent  iterations. 

a.  If  given  the  opportunity  to  improve  a  single  aspect  of  the  documentation,  what 
would  it  be?  And  how? 


Materiel 

5.  Please  describe  your  experience  with  using  the  current  CEE  in  helping  you  go  through 
the  activities  and  production  of  deliverables  thus  far. 

a.  What  tools  worked  as  desired? 

b.  What  tools  were  not  available/did  not  work  as  desired? 

6.  Describe  your  experiences  in  terms  of  exchanging  information  (work  products)  between 
tools  (either  as  individuals  or  as  groups  of  individuals)? 

a.  Were  there  any  interoperability  problems?  Were  they  solved? 

b.  Were  most  issues  addressed  with  technology  or  manual  process? 

7.  Based  on  your  experiences  to  date,  what  challenges  and  issues  do  you  envision  in  a  move 
to  working  at  a  classified  level? 

a.  In  terms  of  information  management? 

b.  In  terms  of  exchanging  information? 

c.  In  terms  of  work  environments? 


People 

8.  Please  identify  any  issues  that  you  may  have  encountered  with  the  specified  roles  and 
responsibilities  of  the  CET. 
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a.  Were  role  names  an  issue  (e.g.,  CET  Coordinator)?  If  so,  how? 

b.  What  was  your  experience  with  contractor  support  to  the  CET? 

c.  To  what  degree  did  SMEs  participate?  What  factors,  if  any,  impacted  SME 
participation? 

9.  At  this  point  in  time,  how  would  you  assess  the  leadership  within  the  CET? 

10.  Please  assess  how  well  the  CET  communicated  throughout  the  Comprehension  Stage. 

1 1 .  Please  describe  how  you  incorporated  the  principles  of  teamwork  and  collaboration 
within  the  CET. 

a.  Describe  how  the  team  worked  together  to  produce  artefacts. 

Evaluation  Team 

12.  Please  describe  the  level  of  support  received  from  the  Evaluation  Team. 

a.  What  went  well? 

b.  What  improvements,  if  any,  would  you  recommend? 

c.  Is  there  a  particular  kind  (style)  of  support  you  would  prefer  from  the  Evaluation 
Team? 

FLA  Signatory  Review  Briefing 

13.  How  would  you  assess  the  PLA  Signatory  Meeting  this  past  Monday,  December  18, 
2006. 

a.  Do  you  feel  Exercise  Gamma  is  “on  track”? 

b.  What  adjustments,  if  any,  would  you  recommend? 

a.  For  example:  issues  of  focus  (content  or  approach),  classification, 
industrial  engagement/involvement,  etc. 
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B.2  Mind  Map 
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Annex  C  Third  Focus  Group 


C.1  Questionnaire 


Focus  Group  Discussion  Guide 
Exercise  Gamma:  Solving  a  Real  Problem 

Exercise  Gamma  is  intended  to  be  a  complete  “3rd  party”  functional  test  and  evaluation  of 
CapDEM’s  CEP,  CET  and  CEE  axes.  Based  on  a  realistic  problem  definition,  Exercise  Gamma 
shifts  emphasis  towards  external  groups  being  able  to  address  their  problem  using  the  CapDEM 
approach.  The  CET  will  be  composed  of  individuals  from  relevant  client  groups.  The  primary 
goal  of  this  iteration  is  to  test  and  evaluate  the  CEP  via  an  instance  based  on  Force  Planning 
Scenario  2. 

To  help  understand  how  Exercise  Gamma  is  progressing  after  completing  the  Comprehension 
Stage,  you  are  being  asked  to  participate  in  this  focus  group  session.  This  focus  group  will  be 
asked  a  variety  of  questions  (along  with  probing  questions)  to  help  the  Evaluation  Team  assess 
the  effectiveness  of  the  three  CapDEM  axes  and  Evaluation  Team. 

At  this  point,  1  would  like  to  reiterate  the  background  of  the  Comprehension  Stage,  including  the 
objectives,  main  tasks  and  milestones. 
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The  CE  Initiative  -  Comprehension  Stage 


Objectives 

The  objectives  of  the  Comprehension  Stage  are  to  develop  and  validate  viable  operational 
options,  to  identify  and  refine  corresponding  operational  requirements,  to  develop  trade  study 
models  and  to  identify  a  preliminary  set  of  candidate  SoS  options. 

Main  tasks 

•  Analyze  scenarios 

•  Outline  operational  options 

•  Refine  operational  options 

Identify  operational  activity  interfaces 
Identify  organizational  composition 
Perform  operational  activity  decomposition 
Analyse  operational  activity  behaviour 

•  Assess  operational  options 

•  Analyze  existing  systems 

•  Identify  potential  systems 

•  Group  and  allocate  systems 

•  Identify  SoS  options 

•  Refine  CEDF  Model  &  Performance  parameter  criteria 

Milestones 

At  the  end  of  the  Comprehension  Stage,  a  first  set  of  viable  operational  options  is  approved, 
feedback  is  received  on  the  set  of  preliminary  SoS  options  and  the  work  plan  for  the  Elaboration 
Stage  is  approved. 
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Focus  Group  Questions 
Comprehension  Stage 


General 

1.  Overall,  how  would  you  assess  your  participation  within  the  Comprehension  Stage? 

Process 

2.  Based  on  the  work  that  you  have  carried  out  within  the  CET  so  far,  how  would  you  assess  the 
CEP? 

3.  How  would  you  assess  the  time  that  it  took  you  to  perform  the  tasks  in  the  Comprehension 
Stage? 

a.  Did  you  experience  any  difficulties  with  respect  to  completing  the  tasks?  If  so, 
can  you  describe  these  difficulties? 

b.  Did  you  experience  any  delays  with  respect  to  completing  the  tasks?  If  so,  can 
you  describe  what  kind  of  delays? 

4.  Given  the  CEP  workflow  for  the  Comprehension  Stage  (see  provided  outline),  please  answer 
the  following: 

a.  What  tasks  have  you  completed? 

b.  How  well  have  you  completed  them? 

c.  Was  the  ordering  of  the  tasks  coherent? 

d.  Were  the  tasks  well-described  and  understandable? 

e.  How  closely  did  you  follow  the  workflow  (as  presented)? 

f.  Did  you  augment  the  workflow?  If  so,  how? 

g.  Would  a  second  iteration  have  influenced  the  outcome  of  this  Stage?  If  so,  how? 

5.  Please  identify  any  process  strengths  (including  deliverable  templates,  online  documentation, 
etc.)  that  would  facilitate  its  application  in  subsequent  iterations. 

a.  What  was  your  experience  with  the  online  CEP  documentation  (i.e.,  the  Web 
page)? 

6.  Please  identify  any  process  weaknesses  (including  deliverable  templates,  online 
documentation,  etc.)  that  would  facilitate  its  application  in  subsequent  iterations. 

a.  If  given  the  opportunity  to  improve  a  single  aspect  of  the  documentation,  what 
would  it  be?  How  would  you  improve  it? 

Materiel 

7.  Please  describe  your  experiences  with  using  the  current  CEE  (mainly  ACCESS  Labs, 
Livelink,  CORE,  Phoenix,  DOORS)  in  helping  you  to  go  through  the  activities  and 
production  of  deliverables  thus  far. 

a.  To  what  extent  did  you  exploit  the  above  tools? 

b.  Which  tools  worked  as  desired? 

c.  Which  tools  were  not  available/did  not  work  as  desired? 

8.  Describe  your  experiences  in  terms  of  exchanging  information  (work  products)  between  tools 
(either  as  individuals  or  as  groups  of  individuals)? 

a.  Were  there  any  interoperability  problems?  Were  they  solved? 

b.  Were  most  issues  addressed  with  technology  or  a  manual  process? 
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9.  Based  on  your  participation  within  the  CET  to  date,  have  you  experienced  any  challenges 
and/or  issues  in  working  at  a  classified  level? 

a.  In  terms  of  information  management? 

b.  In  terms  of  exchanging  information? 

c.  In  terms  of  work  environments? 


People 

10.  Please  describe  how  you  incorporated  the  principles  of  teamwork  and  collaboration  within  the 
CET. 

a.  Describe  how  the  team  worked  together  to  produce  artefacts. 

1 1.  At  this  point  in  time,  how  would  you  assess  the  leadership  within  the  CET? 

12.  Please  assess  how  well  the  CET  communicated  throughout  the  Comprehension  Stage. 

13.  Please  identify  any  issues  that  you  may  have  encountered  with  the  specified  roles  and 
responsibilities  of  the  CET. 

a.  Do  you  feel  that  people  carried  out  their  roles  and  responsibilities  adequately? 

14.  Please  identify  any  issues  that  you  may  have  encountered  with:  contactors  outside  the  CET, 
Operational  Reps  and  PRICIE  Reps. 

a.  What  was  your  experience  with  contractor  support  to  the  CET? 

b.  To  what  extent  did  Operational  Reps  participate? 

i.  What  factors,  if  any,  may  have  impacted  Operational  Rep  participation? 

ii.  If  they  did  not  participate,  do  you  feel  that  it  would  have  been  more 
beneficial  should  they  have  had  participated? 

a.  To  what  extent  did  PRICIE  Reps  participate? 

iii.  What  factors,  if  any,  may  have  impacted  PRICIE  Rep  participation? 

iv.  If  they  did  not  participate,  do  you  feel  that  it  would  have  been  more 
beneficial  should  they  have  had  participated? 

Evaluation  Team 


15.  Please  describe  the  level  of  support  received  from  the  Evaluation  Team. 

a.  What  went  well? 

b.  What  improvements,  if  any,  would  you  recommend? 

16.  How  would  you  assess  the  level  of  coaching  received  since  January  2007? 

a.  What  went  well? 

b.  What  improvements,  if  any,  would  you  recommend? 

Elaboration  Training 


17.  In  comparison  to  previous  training  sessions  (i.e.,  Inception  and  Comprehension  stages),  how 
would  you  assess  the  Elaboration  training? 

a.  What  went  well? 

b.  What  improvements,  if  any,  would  you  recommend? 

c.  Was  the  scheduling  of  the  Elaboration  training  appropriate? 

d.  Was  the  format  for  the  Elaboration  training  appropriate? 
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Annex  D  Fourth  Focus  Group 


D.1  Questionnaire 


Focus  Group  Discussion  Guide 
Exercise  Gamma:  Solving  a  Real  Problem 

Exercise  Gamma  is  intended  to  be  a  complete  “3rd  party”  functional  test  and  evaluation  of 
CapDEM’s  CEP ,  CET  and  CEE  axes.  Based  on  a  realistic  problem  definition,  Exercise  Gamma 
shifts  emphasis  towards  external  groups  being  able  to  address  their  problem  using  the  CapDEM 
approach.  The  CET  will  be  composed  of  individuals  from  relevant  client  groups.  The  primary 
goal  of  this  iteration  is  to  test  and  evaluate  the  CEP  via  an  instance  based  on  Force  Planning 
Scenario  2. 

To  help  understand  how  Exercise  Gamma  is  progressing  after  completing  the 
Elaboration/Completion  Stages,  you  are  being  asked  to  participate  in  this  focus  group  session. 
This  focus  group  will  be  asked  a  variety  of  questions  (along  with  probing  questions)  to  help  the 
Evaluation  Team  assess  the  overall  completion  of  Exercise  Gamma. 
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Focus  Group  Questions 
Post  Elaboration/Completion  Stage 


General 

1.  Overall,  how  would  you  assess  your  participation  within  the  CET  since  the  last  focus  group? 

People  Axis 

2.  Please  identify  the  main  issues  that  you  may  have  encountered  within  the  CET. 

a.  Team  collaboration 

b.  Team  communication 

c.  Roles  and  responsibilities 

d.  Other  issues 

3.  Please  describe  the  elements  you  would  incorporate  in  the  creation  of  the  future  “ideal”  CET. 

4.  Please  describe  what  actions  you  would  take  to  achieve  (i.e.,  move  towards)  the  realization  of 
this  “ideal”  CET. 

Process  Axis 

5.  Please  identify  the  main  issues  that  you  may  have  encountered  within  the  CEP. 

a.  Stages 

b.  Activities/tasks 

c.  Iterations 

d.  Deliverables 

6.  In  general,  how  would  you  assess  your  level  of  satisfaction  with  respect  to  the  completion  of 
the  deliverables? 

a.  Did  you  recommend  one  option?  If  so,  how  did  you  come  to  a  consensus  on  your 
recommendation? 

b.  What  is  your  level  of  confidence  with  respect  to  the  recommended  option  [very 
confident,  confident,  somewhat  confident,  not  confident  at  all]? 

c.  What  are  the  most  relevant  deliverables  which  you  would  recommend  for  the 
future? 

7.  Please  describe  the  elements  you  would  incorporate  in  the  creation  of  the  future  “ideal”  CEP. 

8.  Please  describe  what  actions  you  would  take  to  achieve  (i.e.,  move  towards)  the  realization  of 
this  “ideal”  CEP. 

Materiel  Axis 

9.  Please  identify  the  main  issues  that  you  may  have  encountered  within  the  CEE. 

a.  Functionalities 

b.  Tools 
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c.  Facilities 

d.  Connectivity  and  access 

10.  Please  describe  the  elements  you  would  incorporate  in  the  creation  of  the  future  “ideal”  CEE. 

11.  Please  describe  what  actions  you  would  take  to  achieve  (i.e.,  move  towards)  the  realization  of 
this  “ideal”  CEE. 

Support  Team 

12.  What  kind  of  support  would  be  required  for  future  CET? 

a.  Characteristics  of  a  future  support  team. 

b.  Functionalities 

Future  Feedback  mechanisms 

13.  How  would  you  assess  the  use  of  focus  groups  throughout  Exercise  Gamma? 

14.  Would  you  recommend  any  other  feedback  mechanisms?  If  so,  which  ones  would  you 
recommend? 
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List  of  symbols/abbreviations/acronyms/initialisms 


ACCESS 

Advanced  Collaborative  Capability  Engineering  Support  System 

ADM(IM) 

Assistant  Deputy  Minister  (Information  Management) 

aka 

Also  Known  As 

CAN/US 

Canada/United  States 

CapDEM 

Collaborative  Capability  Definition,  Engineering  and  Management 

CBP 

Capability-Based  Planning 

CCA 

Centre  for  Capability  Analysis 

CE 

Capability  Engineering 

CEE 

Collaborative  Engineering  Environment 

CEO 

Canadian  Eyes  Only 

CEP 

Capability  Engineering  Process 

CET 

Capability  Engineering  Team 

CF 

Canadian  Forces 

CFD 

Chief  of  Force  Development 

CM 

Configuration  Management 

CMSC 

Capability  Management  Support  Centre 

CORA 

Centre  for  Operations  Research  and  Analysis 

COS(IM) 

Chief  of  Staff  (Information  Management) 

COTS 

Commercial  Off-the-Shelf 

CMWG 

Capability  Management  Working  Group 

CSPS 

Canada  School  of  Public  Service 

C4ISR 

Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance 
and  Reconnaissance 

C4+I 

Command,  Control,  Communications,  Computers  and  Information 

DND 

Department  of  National  Defence 

DoDAF 

Department  of  Defence  Architecture  Framework 

DRDC 

Defence  Research  and  Development  Canada 

DRDKIM 

Director  Research  and  Development  Knowledge  and  Information 

Management 

DRENet 

Defence  Research  Establishment  Network 
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DWAN 

Defence  Wide  Area  Network 

ECS 

Environmental  Chief  of  Staff 

EMP 

Engineering  Management  Plan 

NWP 

Northwest  Passage 

IM 

Information  Management 

IT 

Information  Technology 

MSOC 

Marine  Security  Operations  Centre 

PLA 

Project  Level  Agreement 

PM 

Project  Manager 

PRICIE 

Personnel;  Research  and  Development;  Infrastructure  and  Organization; 
Concepts,  Doctrine  and  Collective  Training;  Information  Management; 
Equipment,  Supplies  and  Services 

PSEPC 

Public  Safety  and  Emergency  Preparedness  Canada 

R&D 

Research  and  Development 

SCA 

Strategic  Context  Analysis 

SFA 

Strategic  Factor  Analysis 

SME 

Subject  Matter  Expert 

SoS 

System  of  Systems 

TD 

Technology  Demonstrator 

TDP 

Technology  Demonstration  Programme 

USB 

Universal  Serial  Bus 

VPN 

Virtual  Private  Network 

VTC 

Video  Teleconference 
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problem’  while  still  enabling  the  observation  and  study  of  its  application  within  an  increasingly 
operational-like  environment.  As  the  third  and  final  exercise.  Exercise  Gamma  was  the  largest  and  most 
ambitious  of  the  evaluation  efforts,  the  result  of  gradual,  controlled  growth  from  one  exercise  to  another, 
benefiting  from  the  accumulated  experience  of  the  evaluation  team  along  with  its  validated  evaluation 
strategy.  The  exercise  shifted  away  from  incrementally  controlled  experimentation  towards  the  reality  of 
actual  departmental  clients  applying  Capability  Engineering  and  CapDEM’ s  ability  to  meet  those  needs. 
Accordingly,  this  report  summarizes  the  results  of  Exercise  Gamma  which  was  undertaken  to  evaluate  the 
three  fundamental  axes  that  compose  the  Capability  Engineering  (CE)  construct  (i.e..  People,  Process  and 
Materiel).  Specifically,  the  report  outlines  the  conduct  of  Exercise  Gamma,  the  results  of  observations 
and  focus  groups  that  were  conducted  throughout  the  exercise,  and  provides  discussion  and 
recommendations  to  consider  in  terms  of  the  potential  institutionalization  of  the  CapDEM  approach 
within  the  department. 

L’exercice  Gamma  a  ete  concu  pour  etre  une  epreuve  et  une  evaluation  tout  a  fait  independante  de 
l’approche  axee  sur  la  definition,  l’ingenierie  et  la  gestion  collaboratives  des  capacites  (DIGCap). 
L’objectif  principal  de  ce  dernier  volet  de  la  Strategie  devaluation  de  l’approche  DIGCap  consistait  a 
mettre  a  l’essai  et  a  evaluer  cette  derniere  a  l'aide  d’un  «  probleme  reel  »  fonde  sur  un  scenario  defini  par 
le  Ministere,  l'approche  etant  alors  mise  en  oeuvre  par  une  equipe  composee  de  membres  du  MDN  et  des 
FC.  L’intention  etait  de  valider  les  personnes,  le  processus  et  le  materiel  necessaires  pour  s’attaquer  a  un 
«  probleme  reel  »,  tout  en  permettant  l’observation  et  P  etude  de  1’ application  du  processus  dans  un 
contexte  a  caractere  de  plus  en  plus  operationnel.  En  tant  que  le  troisieme  et  dernier  exercice, 
l’exercice  Gamma  a  ete  le  plus  vaste  et  le  plus  ambitieux  de  tous  les  efforts  d’ evaluation;  il  resultait  de  la 
croissance  graduelle  et  controlee  s’etant  produite  d’un  exercice  a  l’autre  et  il  a  beneficie  de  P  experience 
cumulative  acquise  par  1' equipe  d’ evaluation  et  de  sa  strategie  d’ evaluation  validee.  L’ Exercice  s’est 
eloigne  des  experiences  progressivement  controlees  pour  evoluer  vers  la  realite  de  veritables  clients 
ministeriels,  en  appliquant  l’ingenierie  des  capacites  et  l’outil  DIGCap  pour  repondre  aux  besoins  de  ces 
derniers.  Par  consequent,  le  present  rapport  resume  les  resultats  de  l’exercice  Gamma,  qui  a  ete  entrepris 
pour  evaluer  les  trois  axes  fondamentaux  du  concept  structurel  de  l’ingenierie  des  capacites  (IC) :  les 
personnes,  le  processus  et  le  materiel.  Le  rapport  decrit  P execution  de  l’exercice  Gamma  et  les  resultats 
des  observations  faites  pendant  l’Exercice  et  des  interventions  des  groupes  temoins;  il  offre  une 
discussion  et  des  recommandations  a  etudier  relativement  a  l’institutionnalisation  eventuelle  de 
l’approche  DIGCap  au  sein  du  Ministere. 
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